Python中存储大量自定义类实例的高效容器方案咨询

Great question! The array module works wonders for basic data types because it uses contiguous, memory-efficient storage—but it can't handle custom class instances directly, since each instance is a full Python object with extra overhead beyond its core data. Below are practical solutions tailored to different priorities (speed, memory, convenience):

1. Basic Python List (Simplest Approach) #

If your dataset isn't extremely large (e.g., under 10 million instances) and you need full access to your Point class's methods and properties, a standard list is the easiest way to go:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

# Create a list of 1 million Point instances
points = [Point(0.0, 0.0) for _ in range(10**6)]

Pros: #

• Intuitive and requires no extra libraries
• Preserves all functionality of your custom Point class
• Easy to iterate over and modify instances

Cons: #

• High memory overhead: Each Point instance is a separate Python object with metadata, so memory usage grows quickly with large datasets
• Slower access compared to contiguous memory structures

2. Split Attributes into Separate array Arrays (Most Memory-Efficient) #

This approach mirrors the efficiency of the array module by storing each attribute of your Point class in its own typed array. It's the closest you can get to the original array performance while working with custom data:

from array import array

# Store x and y coordinates in separate float arrays (100 million elements each)
x_coords = array('f', (0.0 for _ in range(10**8)))
y_coords = array('f', (0.0 for _ in range(10**8)))

# Access the i-th "Point" by indexing both arrays
i = 42
print(f"Point {i}: x={x_coords[i]}, y={y_coords[i]}")

# Optional: Convert to a Point instance when needed
def get_point(i):
    return Point(x_coords[i], y_coords[i])

Pros: #

• Near-identical memory efficiency to the original array module (contiguous storage of basic types)
• Blazing-fast access times for attribute values
• Minimal overhead even for hundreds of millions of entries

Cons: #

• Doesn't store actual Point instances—you'll need to index across arrays to access related attributes
• Requires extra code if you frequently need to convert attribute pairs to full Point objects

3. NumPy Structured Arrays (Best for Numerical Workloads) #

If you're working with numerical data and want the benefits of contiguous storage plus vectorized operations, NumPy's structured arrays are a great fit. You can define a dtype that matches your Point class's attributes:

import numpy as np

# Define a structured dtype matching the Point class's x and y attributes
point_dtype = np.dtype([('x', 'f4'), ('y', 'f4')])

# Create an array of 100 million "point" entries
points_array = np.zeros(10**8, dtype=point_dtype)

# Access attributes directly via the structured array
i = 100
print(f"Point {i}: x={points_array[i]['x']}, y={points_array[i]['y']}")

# Optional: Convert a single entry to a Point instance
def to_point(arr_element):
    return Point(arr_element['x'], arr_element['y'])

Pros: #

• Memory-efficient contiguous storage
• Supports NumPy's powerful vectorized operations (great for calculations on coordinates)
• Easy to integrate with other numerical libraries

Cons: #

• Requires NumPy as a dependency
• Like the split array approach, you'll need to convert entries to Point instances if you need full class functionality

Final Recommendation #

• Use a list if you need full Point instances and your dataset size is manageable
• Use split array arrays if you prioritize maximum memory efficiency and speed
• Use NumPy structured arrays if you're doing numerical computations with the data

内容的提问来源于stack exchange，提问作者ArthurQ