如何高效替换NumPy数组的vstack与concatenate组合操作？

Great question—this is a super common pitfall with numpy when dealing with large numbers of iterations! The core issue here is that every call to np.vstack() or np.concatenate() creates a brand new array, which involves expensive memory allocation and data copying. Do this millions of times, and those tiny overheads snowball into a major performance bottleneck.

Here are the most effective fixes, ordered by efficiency:

1. 预分配完整数组，直接索引赋值（最优方案） #

If you know the final shape of your output array upfront (which you should, if you're running millions of consistent iterations), preallocate the entire array once and then write your results directly to the correct slice. This eliminates all repeated memory allocation and copying.

For example, let's say each iteration produces two arrays arr1 (shape (N,)) and arr2 (shape (M,)), and you want to combine them into a row in a 2D result array:

# 先确定最终数组的形状：1e6行，每行长度N+M
total_iterations = 1_000_000
row_length = len(arr1) + len(arr2)
result = np.empty((total_iterations, row_length), dtype=arr1.dtype)

for i in range(total_iterations):
    arr1 = your_first_function()
    arr2 = your_second_function()
    # 直接写入对应位置，避免concatenate
    result[i, :len(arr1)] = arr1
    result[i, len(arr1):] = arr2

This skips both np.concatenate() and np.vstack() entirely—you're just filling preallocated memory, which is as fast as it gets.

2. 用列表收集小数组，最后一次性拼接（次优方案） #

If you can't preallocate (e.g., you don't know the exact final shape upfront), collect all your small arrays in a Python list first, then do a single np.vstack() or np.concatenate() at the end. Python list appends are amortized O(1) and don't involve copying array data, so this reduces the number of expensive memory operations from millions to one.

Example:

temp_list = []
for _ in range(1_000_000):
    arr1 = your_first_function()
    arr2 = your_second_function()
    combined = np.concatenate([arr1, arr2])
    temp_list.append(combined)

# 最后一次性拼接所有结果
result = np.vstack(temp_list)

This is way faster than your original approach because you only pay the cost of one large memory allocation and copy, not millions of small ones.

3. 用Numba加速循环（如果迭代内有复杂计算） #

If your iteration involves more than just collecting arrays (e.g., custom calculations to generate arr1/arr2), wrapping your loop with Numba can compile it to optimized machine code, which can drastically speed up the process—especially when combined with preallocation.

Example:

from numba import jit

@jit(nopython=True)
def fill_result(result):
    for i in range(result.shape[0]):
        # 在这里生成arr1和arr2（Numba支持大部分numpy操作）
        arr1 = some_numba_compatible_function()
        arr2 = another_numba_compatible_function()
        result[i, :len(arr1)] = arr1
        result[i, len(arr1):] = arr2

# 预分配数组
result = np.empty((1_000_000, row_length), dtype=np.float64)
fill_result(result)

Why your original approach was slow #

Every time you call np.vstack(result, new_row), numpy has to:

1. Allocate a new array that's one row larger than result
2. Copy all existing data from result into the new array
3. Copy the new row into the end
4. Discard the old result array

Do this 1,000,000 times, and you're copying O(n²) data total—this is a classic quadratic time bottleneck.

内容的提问来源于stack exchange，提问作者Gabriel