使用Jupyter Notebook运行Kmeans算法时出现BSOD及内存占用疑问

Is 2.5GB Memory Usage Normal? #

Short answer: Yes, it’s reasonable, but there’s clear room for optimization.

Here’s the breakdown:

• Each SIFT descriptor is 128-dimensional. When converted to int32, that’s 128 * 4 bytes = 512 bytes per descriptor. For 2 million descriptors, raw data alone is ~1GB.
• Storing this in a Pandas DataFrame adds significant overhead: DataFrames include indexes, metadata, and use column-based storage that’s less efficient for raw numerical data. The extra 1.5GB comes from these overheads, which is typical for large datasets in Pandas.

Quick Optimization to Cut Memory Usage #

Swap the Pandas DataFrame for a NumPy array—it’s far more memory-efficient for numerical data. Modify your code like this:

import numpy as np

def get_descriptors(df):
    descriptor_list = []
    for img in df.img:
        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        _, descriptor = sift.detectAndCompute(img, None)
        if descriptor is not None:  # Skip images with no keypoints
            descriptor_list.append(descriptor.astype('int32'))
    # Stack all descriptors into a single NumPy array
    return np.vstack(descriptor_list)

descriptor_array = get_descriptors(df)

This should bring your memory usage closer to the raw 1GB mark, eliminating most of the DataFrame overhead.

Fixing the WIN32K_POWER_WATCHDOG_TIMEOUT BSOD #

That error usually triggers when your system runs out of critical resources (memory/CPU) and becomes unresponsive long enough to trigger Windows’ power watchdog. Here’s how to resolve it:

1. Use MiniBatchKMeans Instead of Standard KMeans #

The standard KMeans from scikit-learn loads all data into memory and uses intensive computations that can overwhelm your system. MiniBatchKMeans processes small batches of data at a time, drastically reducing memory usage and CPU load:

from sklearn.cluster import MiniBatchKMeans

# Initialize with 500 clusters and a reasonable batch size
kmeans = MiniBatchKMeans(n_clusters=500, batch_size=10000, random_state=42)
kmeans.fit(descriptor_array)

This prevents system overload while still delivering good clustering results.

2. Adjust WSL Resource Limits #

By default, WSL 2 allocates up to 50% of your host’s RAM (8GB in your case) and all CPU cores. Tweak these limits to avoid starving your Windows host:

• Create a .wslconfig file in your Windows user directory (C:\Users\<YourUsername>\.wslconfig) with these settings:

  [wsl2]
  memory=10GB  # Allocate more RAM to WSL (leave 6GB for Windows)
  processors=3  # Limit to 3 CPU cores so Windows stays responsive
  

• Shut down WSL with wsl --shutdown in Command Prompt, then restart it.

3. Reduce Computational Load #

• Dimensionality Reduction with PCA: Use PCA to shrink the 128-dimensional SIFT descriptors to a smaller size (e.g., 64). This cuts memory usage and speeds up clustering:

  from sklearn.decomposition import PCA
  
  pca = PCA(n_components=64, random_state=42)
  reduced_descriptors = pca.fit_transform(descriptor_array)
  # Now run MiniBatchKMeans on reduced_descriptors
  

• Test with Fewer Clusters: Start with 200 clusters instead of 500 to see if the BSOD goes away. You can gradually increase the number once you have a stable setup.

4. System-Level Fixes #

• Update Windows: Your Windows 10 version (18362) is outdated. Updating to the latest stable version will fix known WSL stability bugs and improve resource management.
• Close Background Apps: Shut down unnecessary programs (browsers, video editors, etc.) to free up RAM and CPU for your task.

内容的提问来源于stack exchange，提问作者Célia Bayet