关于超4GB大型Numpy数组、npy文件及memmap的技术问询

Let's break down your questions one by one, based on the NPY format specs and NumPy's actual behavior:

1. What exactly does "header size" refer to in NPY files? #

The header in an NPY file is the section that stores metadata about the NumPy array, not the raw array data itself. This metadata includes critical details like:

• The array's shape (e.g., (5000, 5000) for a 2D array)
• Data type (e.g., float32, int64, or custom structured dtypes)
• Byte order (little-endian vs. big-endian)
• Memory layout flags (C-ordered vs. Fortran-ordered)

Before NPY format version 2.0, the header had a strict size limit that restricted how much metadata could be stored. The 2.0 update expands this limit to 4 GiB, which means you can now handle extremely complex arrays—like structured arrays with thousands of fields, or arrays with an unusually large number of dimensions—without hitting header size constraints.

2. Does this mean NPY files can only store arrays up to 4 GB? #

Absolutely not. The 4 GiB limit applies only to the metadata header, not the actual array data. An NPY file is split into two distinct parts: the header (metadata) and the data block (the raw bytes of the array). The header is usually just a few kilobytes even for large arrays, since it only needs to describe the array's properties. The total size of the NPY file (and thus the array it stores) can be as large as your storage device and system resources allow.

For example, a 100 GB NumPy array would have a tiny header (maybe 100 bytes) and a 100 GB data block—this is fully supported by the 2.0 format.

3. What's the maximum size of a single NumPy array? #

NumPy itself doesn't enforce a hard size limit—this depends almost entirely on your system's architecture and resources:

• 32-bit systems: Due to memory address space limitations, the maximum practical array size is around 2 GB (since 32 bits can only address ~4 GB of total memory, and a portion of that is reserved for the OS and other programs).
• 64-bit systems: The theoretical limit is enormous (up to 1 exabyte, or 10¹⁸ bytes), but in practice, it's constrained by your available physical memory (RAM) plus virtual memory (swap space). If you try to create an array larger than your system can handle, you'll hit a MemoryError.

The array's dtype also plays a role: a float64 array uses 8 bytes per element, so an array with 10¹¹ elements would require 800 GB of memory—possible on high-performance computing clusters, but not on most consumer systems.

Format Specification: Version 2.0 states that "Version 2.0 of the format expands the header size to 4 GiB."

内容的提问来源于stack exchange，提问作者user8930103