CUDA内存不足报错：SVM网格搜索代码优化与参数调优咨询

运行基于RAPIDS框架的SVM网格搜索代码时触发std::bad_alloc: out_of_memory CUDA错误，3168次模型拟合全部失败，寻求以下帮助：

1. 如何优化代码解决GPU内存不足问题？未找到GridSearchCV设置可用内存的选项；
2. 使用如此庞大的参数网格进行模型调优是否为最佳实践？是否有更常用的替代策略？

硬件与软件配置 #

• 硬件：11代Intel Core i5-11400 @2.60GHz，NVIDIA GeForce RTX 3080 Ti（12.88GB可用显存）
• 软件：RAPIDS 24.02，Python 3.10（Spyder环境）

代码 #

import cudf
import cuml
import cupy as cp
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from cuml.model_selection import GridSearchCV
from cuml.metrics import accuracy_score
import pandas as pandas
#---------------------------------------------------------------------------------------------------
def preparaDatos(y,pathMetadata,pathMatrizUnitigs) :
        
    print("Preparing data for modeling...")
    y_column = pandas.read_csv(pathMetadata,index_col=0,delimiter=';')
    y_column = y_column.dropna(subset=[y]) 
    y_column = y_column[y]
      
    X = pandas.read_csv(pathMatrizUnitigs, sep="\t", index_col=0, low_memory=False)
    X = X.transpose()
    X = X[X.index.isin(y_column.index)] 
    
    y_column = y_column[y_column.index.isin(X.index)]
        
    ordered_indices = y_column.index
   
    X_ordered = X.loc[ordered_indices]

    return X_ordered, y_column

#---------------------------------------------------------------------------------------------------
def main():
    espacioTrabajo = ""
    archUnitigs = "graph.nodes"
    archMatrizUnitigs = "salida_Unitig_caller.rtab"
    archMetadata = "metadata.csv"
    archMatrizUnitigsFiltrada = "salida_Unitig_caller_gt90.txt"
    pathArchUnitigs = espacioTrabajo + archUnitigs
    pathMetadata = espacioTrabajo + archMetadata
    pathArchMatrizUnitigsFiltrada = espacioTrabajo + archMatrizUnitigsFiltrada


        
    classField="VC"
    X, y = preparaDatos(classField,pathMetadata,pathArchMatrizUnitigsFiltrada)

    
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
   
    X_train_cudf = cudf.DataFrame(X_train)
    X_test_cudf = cudf.DataFrame(X_test)
    y_train_cudf = cudf.Series(y_train)
    y_test_cudf = cudf.Series(y_test)
    
 
    X_train_cudf = X_train_cudf.astype('float32')
    y_train_cudf = y_train_cudf.astype('float32')
    
    # 定义网格搜索参数
    params = {
             'C': [0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 1, 1.3, 1.5, 1.7, 2.0, 3.0, 4,0, 5.0, 6.0, 7.0, 8.0, 9.0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100],
             'gamma': [0.000001, 0.00001, 0.0001, 0.001, 0.01, 0.1, 1, 10],
             'kernel': ['linear', 'poly' , 'rbf' , 'sigmoid']
     }
    
   
    svm_model = cuml.svm.SVC(cache_size=10240)
    
    
    grid_search = GridSearchCV(estimator=svm_model, param_grid=params, cv=3, n_jobs=-1)
    
    print("Performing GRID search...")
   
    grid_search.fit(X_train_cudf.to_numpy(), y_train_cudf.to_numpy())
    
    best_params = grid_search.best_params_
    
    print("Best params: ")
    print(best_params)
    
    #----------------------------------------------------------------------------------------------------
    
if __name__ == "__main__":
    main()

错误信息 #

Traceback (most recent call last):

  File ~/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File ~/Documentos/ML/Python/Pruebas/SVM_GPU_6.py:106
    main()

  File ~/Documentos/ML/Python/Pruebas/SVM_GPU_6.py:85 in main
    grid_search.fit(X_train_cudf.to_numpy(), y_train_cudf.to_numpy())

  File ~/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/sklearn/base.py:1474 in wrapper
    return fit_method(estimator, *args, **kwargs)

  File ~/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/sklearn/model_selection/_search.py:970 in fit
    self._run_search(evaluate_candidates)

  File ~/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/sklearn/model_selection/_search.py:1527 in _run_search
    evaluate_candidates(ParameterGrid(self.param_grid))

  File ~/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/sklearn/model_selection/_search.py:947 in evaluate_candidates
    _warn_or_raise_about_fit_failures(out, self.error_score)

  File ~/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:536 in _warn_or_raise_about_fit_failures
    raise ValueError(all_fits_failed_message)

ValueError: 
All the 3168 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
3168 fits failed with the following error:
Traceback (most recent call last):
  File "/home/veronica/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 895, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/veronica/anaconda3/envs/rapids-24.02/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper
    ret = func(*args, **kwargs)
  File "svc.pyx", line 545, in cuml.svm.svc.SVC.fit
  File "svc.pyx", line 547, in cuml.svm.svc.SVC.fit
MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /home/veronica/anaconda3/envs/rapids-24.02/include/rmm/mr/device/cuda_memory_resource.hpp

一、GPU内存不足的代码优化策略 #

1. 限制并行任务数
   当前GridSearchCV设置n_jobs=-1会启用所有CPU核心并行运行任务，每个任务都占用GPU显存，直接导致显存耗尽。将n_jobs改为1或2，强制串行或少量并行运行，避免同时加载过多模型到GPU。

2. 优化数据传递方式
   代码中将cudf.DataFrame转成numpy数组再传入fit，会导致数据在GPU与CPU之间来回传输，浪费显存与时间。直接传入cudf对象即可：

   grid_search.fit(X_train_cudf, y_train_cudf)
   

3. 降低SVM缓存大小
   当前设置cache_size=10240（10GB），几乎占满了可用显存。改为更小的值，比如cache_size=2048（2GB），减少单个模型占用的显存：

   svm_model = cuml.svm.SVC(cache_size=2048)
   

4. 修正参数错误
   C参数列表中的4,0是语法错误，应改为4.0，该错误可能导致模型初始化异常，额外消耗显存。

5. 特征降维
   若输入数据特征维度极高（从代码中的unitig矩阵来看大概率如此），用cuml.decomposition.PCA或cuml.feature_selection.SelectKBest做降维，减少特征数量，降低SVM计算时的显存占用。

6. 手动管理GPU显存
   加入RAPIDS的RMM内存管理器限制显存使用，或在关键节点清理缓存：

   from rmm import set_allocator, malloc
   set_allocator(malloc, pool_limit=10*1024**3)  # 限制显存使用为10GB
   # 或在合适位置清理缓存
   cp.clear_memo()
   cudf.clear_cache()
   

二、参数调优的最佳实践与替代策略 #

1. 当前网格的问题
   你的参数网格共有33（C）×8（gamma）×4（kernel）=1056种组合，加上3折交叉验证，总共3168次拟合：

   • 计算效率极低，耗时极长；
   • 存在大量冗余组合，比如linear核不需要gamma参数，完全是无效计算；
   • 并行运行时显存压力直接超出GPU承载能力。

2. 替代策略

   • 分步网格搜索：先粗搜大范围参数，再在最优区域细搜。比如先将C设为[0.01,0.1,1,10,100]，gamma设为[1e-6,1e-4,1e-2,1]，找到大致最优区间后再缩小范围细化。
   • 随机搜索：用cuml.model_selection.RandomizedSearchCV替代GridSearchCV，随机采样参数组合，通常100-200次采样就能找到接近最优的参数，效率远高于全网格搜索。
   • 贝叶斯优化：用cuml.model_selection.BayesianSearchCV，基于之前的搜索结果智能选择下一组参数，比随机搜索更高效，尤其适合高维参数空间。
   • 按核拆分搜索：不同核的参数需求不同，分开搜索减少无效组合：

     # 单独搜索linear核（仅需调C）
     params_linear = {'C': [...], 'kernel': ['linear']}
     # 搜索非线性核（需调C和gamma）
     params_nonlinear = {'C': [...], 'gamma': [...], 'kernel': ['rbf','poly','sigmoid']}
     

内容的提问来源于stack exchange，提问作者Veronica