如何直接创建numpy对象数组以规避列表构造的耗时问题？

Great question—let’s break this down step by step, covering both direct object array tweaks and broader strategies to save time while keeping NumPy’s fast computation benefits.

Can we skip the list comprehension and create a NumPy object array directly? #

Short answer: You can avoid the intermediate Python list, but you can’t get around initializing each Myclass instance individually. Here’s why:

NumPy object arrays store pointers to Python objects under the hood, so each Myclass instance still needs to be constructed one by one. That said, using np.empty to pre-allocate the object array then filling it in a loop can be slightly faster than a list comprehension followed by np.array(), because you skip the step of building a full Python list first.

Example code for this approach:

import numpy as np

# Pre-allocate an empty object array (no Myclass instances initialized yet)
arr = np.empty(10000, dtype=object)

# Fill the array with Myclass instances
for i in range(arr.shape[0]):
    arr[i] = Myclass(np.random.random(100))

This cuts out the overhead of creating a temporary Python list, which becomes more noticeable as the number of instances grows. But keep in mind: the time spent initializing each Myclass object itself remains the same—this only optimizes the array conversion step.

Better Time-Saving Strategies for Your Use Case #

Since your goal is to leverage NumPy’s fast computations (like column sums) on slices combining Myclass objects and scalar fields, object arrays might not be the most efficient choice. Object arrays often fall back to Python-level loops for operations, negating NumPy’s vectorization benefits. Here are better alternatives:

1. Split Myclass Attributes into Separate NumPy Arrays #

Instead of storing full Myclass objects in an array, extract each attribute into its own dedicated NumPy array. For example, if your Myclass looks like this:

class Myclass:
    def __init__(self, float_list):
        self.id = np.random.randint(0, 100)  # Integer attribute
        self.score = np.random.random()      # Float attribute
        self.data = float_list               # Float list/tuple (shape (100,))

You can replace the object array with:

# Batch-generate all attributes as NumPy arrays
ids = np.random.randint(0, 100, size=10000)
scores = np.random.random(size=10000)
data_arr = np.random.random(size=(10000, 100))
extra_scalars = np.random.random(size=10000)  # Your additional scalar fields

Now you can perform all desired computations with pure NumPy vectorization:

• Column sums of data_arr: data_arr.sum(axis=0)
• Sum scores for a slice: scores[500:1000].sum()
• Combine slices across arrays (e.g., get ids + scores where data meets a condition): ids[data_arr[:,0] > 0.5], scores[data_arr[:,0] > 0.5]

This is by far the fastest approach because it eliminates all Python object overhead during computations.

2. Use NumPy Structured Arrays #

If you prefer a more "object-like" structure where related attributes are grouped, a structured array is a great middle ground. It lets you access attributes by name while retaining NumPy’s vectorization speed.

Define a custom dtype matching your Myclass attributes, then populate the array in batches:

# Define the structured dtype
custom_dtype = [
    ('id', 'int32'),
    ('score', 'float64'),
    ('data', '100float64'),  # 100-element float array per entry
    ('extra_scalar', 'float64')  # Your additional scalar field
]

# Create empty structured array
structured_arr = np.empty(10000, dtype=custom_dtype)

# Batch-fill each attribute
structured_arr['id'] = np.random.randint(0, 100, size=10000)
structured_arr['score'] = np.random.random(size=10000)
structured_arr['data'] = np.random.random(size=(10000, 100))
structured_arr['extra_scalar'] = np.random.random(size=10000)

You can access elements just like objects: structured_arr[0]['score'], and run fast vectorized operations: structured_arr['data'].sum(axis=0) or structured_arr[structured_arr['score'] > 0.7]['id'].

3. Optimize Myclass Initialization #

If you must keep using Myclass objects, optimize their creation:

• Batch-generate shared data: Instead of creating a new random array for each Myclass instance, generate one large array upfront and pass slices to each instance. This saves memory allocation time:

  # Pre-generate all data in one batch
  all_data = np.random.random(size=(10000, 100))
  arr = np.empty(10000, dtype=object)
  for i in range(10000):
      arr[i] = Myclass(all_data[i])  # Pass a slice (or view) of the big array
  

• Avoid unnecessary copies: If Myclass doesn’t modify the input float_list, pass it as a view instead of copying it (the above example does this by default if you don’t copy inside Myclass).

Final Takeaway #

• Direct object array creation via np.empty + loop saves a little time over list comprehensions, but doesn’t eliminate per-instance initialization cost.
• For maximum speed with NumPy computations, split attributes into separate arrays or use structured arrays—these let you fully leverage vectorization and avoid Python object overhead.

内容的提问来源于stack exchange，提问作者auser