256GB RAM系统Numpy报ArrayMemoryError与MemoryError解决方案

• 操作系统：Windows 10
• 硬件配置：256GB 运行内存（RAM）、2TB 硬盘
• 开发工具：PyCharm 社区版
• Python 版本：

Python 3.9.7 [MSC v.1929 64 bit (AMD64)] on win32

• 数据集规模：120万行、约7000列
• 触发场景：预处理完成的数据传入Scikit-learn随机森林（Random Forest）模型拟合训练时触发内存报错

joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\joblib\externals\loky\process_executor.py", line 431, in _process_worker
    r = call_item()
  File "C:\Program Files\Python39\lib\site-packages\joblib\externals\loky\process_executor.py", line 285, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "C:\Program Files\Python39\lib\site-packages\joblib\_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 262, in __call__
    return [func(*args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 262, in <listcomp>
    return [func(*args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\fixes.py", line 209, in __call__
    return self.function(*args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_validation.py", line 674, in _fit_and_score
    X_test, y_test = _safe_split(estimator, X, y, test, train)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\metaestimators.py", line 286, in _safe_split
    X_subset = _safe_indexing(X, indices)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\__init__.py", line 377, in _safe_indexing
    return _array_indexing(X, indices, indices_dtype, axis=axis)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\__init__.py", line 201, in _array_indexing
    return array[key] if axis == 0 else array[:, key]
  File "C:\Program Files\Python39\lib\site-packages\numpy\core\memmap.py", line 331, in __getitem__
    res = super(memmap, self).__getitem__(index)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 721. MiB for an array with shape (120000, 7000) and data type uint8
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\...\main.py", line 45, in <module>
    model.fit(X, y)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_search.py", line 891, in fit
    self._run_search(evaluate_candidates)
  File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_search.py", line 1766, in _run_search
    evaluate_candidates(
  File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_search.py", line 838, in evaluate_candidates
    out = parallel(
  File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 1054, in __call__
    self.retrieve()
  File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 933, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Program Files\Python39\lib\site-packages\joblib\_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\Program Files\Python39\lib\concurrent\futures\_base.py", line 445, in result
    return self.__get_result()
  File "C:\Program Files\Python39\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
numpy.core._exceptions.MemoryError: Unable to allocate 721. MiB for an array with shape (120000, 7000) and data type uint8

• 调大PyCharm控制台内存上限
• 增大系统分页文件大小
• 切换运行环境：先后在PyCharm、Jupyter、Google Colab、命令行环境下测试，均触发相同报错

这个报错的核心原因不是总硬件内存不足，是joblib多进程执行交叉验证/参数搜索时，每个子进程会独立拷贝数据副本，叠加交叉验证切分时生成的临时子集数组，内存峰值超过单进程可分配上限，按以下顺序排查：

1. 先验证多进程问题根因
   所有涉及并行的参数统一先设为单进程：模型本身的n_jobs设为1，网格/随机搜索的n_jobs也设为1，关闭多进程后运行代码。如果报错消失，即可确认是多进程内存副本叠加导致的问题。
   参考代码：

   from sklearn.ensemble import RandomForestClassifier
   from sklearn.model_selection import GridSearchCV
   
   # 单进程测试
   rf = RandomForestClassifier(n_jobs=1)
   # 搜索类的n_jobs也设为1，pre_dispatch默认参数先不改动
   search = GridSearchCV(rf, param_grid=param_grid, n_jobs=1, cv=5)
   search.fit(X, y)
   

2. 调整并行参数控制内存峰值
   单进程运行正常后，不要直接用n_jobs=-1占满所有CPU核心，从n_jobs=4开始逐步调大，同时打开任务管理器观察内存占用，找到不触发报错的最大并行数。
   同时给网格/随机搜索接口设置pre_dispatch=2，替代默认的pre_dispatch='2*n_jobs'，限制同时预加载到内存的任务数，避免提前生成多份数据副本占满内存。
3. 从数据层面降低内存开销
   • 先做特征筛选：7000列特征对随机森林模型冗余度极高，先用方差阈值过滤、特征重要性排序等方法筛掉无效特征，把列数降到1000以内，内存占用可直接降到原来的1/7，从根源上减少内存消耗。
   • 稀疏特征转稀疏矩阵：如果特征是大量0值的稀疏类型，把稠密numpy数组转成scipy的CSR稀疏矩阵格式，sklearn随机森林原生支持稀疏矩阵输入，内存占用可降低一个数量级。
4. 确认依赖版本正确性
   虽然安装的是64位Python，也要确认numpy、scikit-learn、joblib都是64位正式版本，没有混装32位依赖。执行以下命令查看已安装包版本，卸载异常版本后重装对应64位包即可：

   pip list | findstr numpy
   pip list | findstr scikit-learn
   pip list | findstr joblib
   

5. 替代方案
   如果以上调整后仍有内存问题，直接换用支持外存训练、内存效率更高的树模型实现，比如LightGBM的随机森林接口、sklearn自带的HistGradientBoosting系列模型，不需要把全量数据常驻内存，训练速度也比sklearn原生随机森林快数倍。

内容的提问来源于stack exchange，提问作者Vance Pyton