Python内存不足解决方案：扩展内存及U盘/硬盘作虚拟内存方法

Got it, let's tackle this problem head-on. That MemoryError when running df_features = df_features.pivot(index='RepID', columns='Code', values='Frequency') makes total sense with only 8GB of RAM—your pivot is trying to create a ~1.34GB float64 array, which is pushing your notebook over the edge, especially when you factor in other memory usage from the OS and Python environment.

Let's split the solutions into two parts: first fixing the pivot-specific memory issue (the better long-term fix), then setting up virtual memory to use your USB/hard drive as extra "RAM" (a temporary workaround).

Part 1: Optimize the Pivot Operation to Save Memory #

These changes will reduce the memory footprint of your data so the pivot can run without hitting the limit.

1. Downcast Numeric Data Types #

The default float64 takes 8 bytes per value—switching to float32 cuts that in half, which would bring your array's memory usage down to ~670MB (well within your 8GB RAM).

# Convert Frequency column to float32 before pivoting
df_features['Frequency'] = df_features['Frequency'].astype('float32')
# Now run the pivot
df_features = df_features.pivot(index='RepID', columns='Code', values='Frequency')

If your Frequency values are integers, you can even use smaller integer types like int32 or int16 if the range allows—check with df_features['Frequency'].describe() first to confirm the value range fits.

2. Use Sparse Matrices for Sparse Data #

If your pivot results in lots of NaN values (which is common in such operations), pandas supports sparse data structures that only store non-null values. This can drastically reduce memory usage:

# Use sparse pivot (pandas 1.0+)
df_features_sparse = df_features.pivot(
    index='RepID', 
    columns='Code', 
    values='Frequency'
).sparse.to_coo()  # Converts to a scipy sparse matrix
# Convert back to a sparse DataFrame if you need pandas functionality
df_features_sparse_df = pd.DataFrame.sparse.from_spmatrix(df_features_sparse)

3. Process Data in Chunks #

Instead of loading the entire dataset into memory, split it into smaller chunks, pivot each chunk, then combine the results. For example, if you're loading from a CSV:

chunk_size = 10000  # Adjust based on your available memory
chunks = []
for chunk in pd.read_csv('your_data.csv', chunksize=chunk_size):
    chunk_pivot = chunk.pivot(index='RepID', columns='Code', values='Frequency')
    chunks.append(chunk_pivot)
# Combine chunks with outer join to keep all RepIDs/Codes
df_features = pd.concat(chunks, axis=0).groupby('RepID').sum()

4. Use Out-of-Core Libraries #

For datasets that are way too big for RAM, use libraries designed to handle out-of-core processing:

• Dask: Works like pandas but processes data in parallel and only loads chunks into memory. Its dask.dataframe has a pivot_table method that mirrors pandas' functionality.
• Vaex: Uses memory-mapped files so you can work with datasets larger than your RAM without loading everything into memory at once.

Part 2: Using USB/Hard Drive as Virtual Memory (Swap Space) #

Virtual memory lets your OS use storage space as if it were RAM. Note: This is slower than physical RAM, so it's a fallback—not a replacement for optimizing your code.

Windows #

1. Right-click This PC → Properties → Advanced system settings.
2. Go to the Advanced tab → click Settings under Performance.
3. In the Performance Options window, go to the Advanced tab → click Change under Virtual memory.
4. Uncheck Automatically manage paging file size for all drives.
5. Select your USB drive or extra hard disk partition → choose Custom size.
6. Set Initial size (MB) to a value like 8192 (8GB) and Maximum size (MB) to 16384 (16GB) (adjust based on your available storage space).
7. Click Set, then OK and restart your computer.

Linux #

1. Connect your USB drive and mount it (e.g., to /mnt/usb).
2. Create a swap file on the drive:

   sudo fallocate -l 16G /mnt/usb/swapfile  # Creates a 16GB swap file
   sudo chmod 600 /mnt/usb/swapfile  # Secure the swap file from unauthorized access
   sudo mkswap /mnt/usb/swapfile  # Format the file as swap space
   sudo swapon /mnt/usb/swapfile  # Activate the swap file
   

3. To make the swap permanent (so it activates on reboot), add this line to /etc/fstab:

   /mnt/usb/swapfile none swap sw 0 0
   

macOS #

1. Connect your USB drive and note its mount point (e.g., /Volumes/MyUSB).
2. Create a sparse disk image for swap:

   sudo hdiutil create -size 16g -type SPARSE -fs HFS+J /Volumes/MyUSB/swap.sparseimage
   

3. Attach the image to your system:

   sudo hdiutil attach /Volumes/MyUSB/swap.sparseimage
   

4. Enable swap for the image:

   sudo sysctl vm.swapfilesize=16777216  # 16GB in KB
   

   Note: macOS manages virtual memory automatically by default, so this adjustment is temporary—you may need to re-run the sysctl command after rebooting.

内容的提问来源于stack exchange，提问作者asmgx