如何更快将二进制文件转为3D numpy数组并优化内存占用？

Great work getting the initial optimization down—let's take this further to cut both memory usage and runtime even more. Your current approach is better than the original, but we can leverage NumPy's vectorized operations and avoid unnecessary data conversions to squeeze out maximum efficiency.

Key Issues in the Current Optimized Code #

1. Unnecessary Float Conversion: Storing pixel data as float immediately doubles/triples memory usage (uint16 is 2 bytes per value; float32 is 4, float64 is 8). You only need float types when calculating means or other statistics—keep raw data in its native uint16 format until you need to compute something.
2. Per-Frame Loop: Even with np.frombuffer, looping over each frame adds overhead. We can load all image data in one go and reshape it directly.
3. np.resize vs np.reshape: resize creates a copy of the array, while reshape just changes how NumPy interprets the memory (O(1) operation, no data copying).

Optimized Implementation (Minimal Memory, Max Speed) #

This approach reads all relevant image data at once, skips headers in a single seek, and keeps data in its native uint16 format:

import numpy as np
import matplotlib.pyplot as plt

filename = 'blabla'
# Image dimensions and frame count
width = 320
height = 256
count_frame = 1300
# Header sizes
main_header_size = 4000
frame_header_size = 100

# Calculate total bytes occupied by all headers
total_header_bytes = main_header_size + count_frame * frame_header_size
# Each frame is width*height uint16 values (2 bytes each)
frame_byte_count = width * height * 2

with open(filename, mode="rb") as f:
    # Jump past all headers in one step
    f.seek(total_header_bytes)
    # Read all image data in a single read operation (fastest way)
    raw_image_data = f.read(count_frame * frame_byte_count)

# Convert raw bytes directly to a uint16 array (little-endian)
# '<u2' is shorthand for little-endian unsigned 16-bit integer
images = np.frombuffer(raw_image_data, dtype='<u2')

# Reshape to (number of frames, height, width) — no data copied!
images = images.reshape(count_frame, height, width)

# If you need float for calculations later, convert only when necessary
# (float32 is usually sufficient for most image stats, saving memory vs float64)
images_float = images.astype(np.float32)

# Verify the first image
plt.figure()
plt.imshow(images[0])
plt.show()

Performance & Memory Improvements #

• Memory Usage: Raw uint16 array takes ~203MB (1300 * 256 * 320 * 2 bytes). Even if converting to float32, it's only ~406MB—way down from your current 1.5GB final usage.
• Runtime: The entire operation should take well under 1 second (no loops, single read and reshape operations are optimized in C under the hood).

Bonus: Memory Mapping for Extremely Large Files #

If you ever work with files too big to fit in memory (e.g., 100k+ frames), use NumPy's memmap to access data directly from disk without loading everything into RAM:

# Memory-mapped array (no data loaded into RAM initially)
images_mmap = np.memmap(
    filename,
    dtype='<u2',
    mode='r',
    offset=total_header_bytes,
    shape=(count_frame, height, width)
)

# Access frames on-demand (loads only the needed data into RAM)
plt.imshow(images_mmap[0])
plt.show()

# Clean up when done
del images_mmap

This keeps memory usage near zero while still letting you work with the array like a regular NumPy array.

内容的提问来源于stack exchange，提问作者Mathieu Gauquelin