为何5维Numpy数组按块打乱后内存占用骤降且形态功能正常？

Great question! This behavior ties together how NumPy handles array indexing, memory continuity, and how Google Colab's memory monitor reports usage. Let's break it down step by step:

1. First, confirm the basics #

Your original array a = np.random.rand(715, 80, 96, 96, 3).astype(np.float32) is a contiguous memory block (verify with print(a.flags.contiguous) — it will return True). It uses ~6GB of memory, which aligns with your calculations.

When you run shuffler = np.random.permutation(a.shape[0]) and a = a[shuffler], you're using integer array indexing (fancy indexing). Unlike basic slicing (which creates a view of the original array), fancy indexing always generates a copy of the data.

2. Why does memory usage appear to drop? #

Here's the core explanation:

• When you reassign a to the shuffled copy, the original contiguous 6GB array loses all references. Python's garbage collector immediately reclaims this large, single memory block, and the operating system marks it as free.
• The new shuffled array is non-contiguous (check a.flags.contiguous now — it will return False). Instead of one big block, it’s made up of 715 smaller, scattered chunks (each corresponding to a shuffled block of 80 images).

Google Colab's memory monitor prioritizes reporting large, contiguous memory allocations. When the original big block is freed, the monitor shows a massive drop in used memory, but it doesn’t immediately account for the scattered smaller chunks of the new array (even though they still add up to ~6GB total).

3. Why reshaping and shuffling doesn’t change memory usage? #

When you reshape the array to (57200,96,96,1), you’re still working with a contiguous array (reshape preserves continuity if the original array was contiguous). Shuffling this with fancy indexing creates another non-contiguous copy, but now it’s made up of 57,200 tiny chunks (~36KB each) instead of 715 medium chunks.

In this case, the OS can’t reclaim a single large block of memory (since the original array’s memory is split into far more small pieces), so the memory monitor shows usage consistent with the original array — which matches your expectation.

4. How to verify this behavior? #

Run these checks to confirm the explanation:

• Print the total bytes of the array before and after shuffling: print(a.nbytes) — it will stay at ~6GB, proving the actual memory usage hasn’t changed.
• Check if the array is contiguous: print(a.flags.contiguous) — this switches from True to False after shuffling.
• Force garbage collection to see immediate changes: import gc; gc.collect() — this ensures the original array is reclaimed right away.

内容的提问来源于stack exchange，提问作者zcb