Ubuntu 20.04带CUDA支持的LightGBM训练报错求助

问题描述 #

我按照官方文档安装了带CUDA支持的LightGBM，尝试过多种安装变体，但训练时始终出现如下致命错误：

[LightGBM] [Fatal] Check failed: (split_indices_block_size_data_partition) > (0) at /home/azureuser/localfiles/LightGBM/lightgbm-python/src/treelearner/cuda/cuda_data_partition.cpp, line 280 .

我找不到问题原因和解决方法，已在conda环境中测试Python 3.10、3.11版本，使用最新版LightGBM和CUDA。猜测CUDA实现比其他GPU实现更快，但不确定。完整报错回溯如下：

{
    "name": "LightGBMError",
    "message": "Check failed: (split_indices_block_size_data_partition) > (0) at /home/azureuser/localfiles/LightGBM/lightgbm-python/src/treelearner/cuda/cuda_data_partition.cpp, line 280 .",
    "stack": "---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
Cell In[2], line 19
     12 # 创建带GPU支持的LightGBM分类器
     13 clf = lgb.LGBMClassifier(
     14     objective='binary',
     15     device='cuda',
     16     verbose=1,
     17 )
---> 19 clf.fit(X_train, y_train)
     21 # 预测并评估
     22 y_pred = clf.predict(X_test)

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/sklearn.py:1421, in LGBMClassifier.fit(self, X, y, sample_weight, init_score, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_metric, feature_name, categorical_feature, callbacks, init_model)
   1418         else:
   1419             valid_sets.append((valid_x, self._le.transform(valid_y)))
-> 1421 super().fit(
   1422     X,
   1423     _y,
   1424     sample_weight=sample_weight,
   1425     init_score=init_score,
   1426     eval_set=valid_sets,
   1427     eval_names=eval_names,
   1428     eval_sample_weight=eval_sample_weight,
   1429     eval_class_weight=eval_class_weight,
   1430     eval_init_score=eval_init_score,
   1431     eval_metric=eval_metric,
   1432     feature_name=feature_name,
   1433     categorical_feature=categorical_feature,
   1434     callbacks=callbacks,
   1435     init_model=init_model,
   1436 )
   1437 return self

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/sklearn.py:1015, in LGBMModel.fit(self, X, y, sample_weight, init_score, group, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_group, eval_metric, feature_name, categorical_feature, callbacks, init_model)
   1012 evals_result: _EvalResultDict = {}
   1013 callbacks.append(record_evaluation(evals_result))
-> 1015 self._Booster = train(
   1016     params=params,
   1017     train_set=train_set,
   1018     num_boost_round=self.n_estimators,
   1019     valid_sets=valid_sets,
   1020     valid_names=eval_names,
   1021     feval=eval_metrics_callable,  # type: ignore[arg-type]
   1022     init_model=init_model,
   1023     callbacks=callbacks,
   1024 )
   1026 # 该代码会填充self.n_features_属性（拟合模型的特征数），因此只能在拟合后设置
   1027 #
   1028 # 相关属性self._n_features_in会填充self.n_features_in_，它在拟合前设置
   1031 self._n_features = self._Booster.num_feature()

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/engine.py:361, in train(params, train_set, num_boost_round, valid_sets, valid_names, feval, init_model, feature_name, categorical_feature, keep_training_booster, callbacks)
    349 for cb in callbacks_before_iter:
    350     cb(
    351         callback.CallbackEnv(
    352             model=booster,
   (...)
    358         )
    359     )
-> 361 booster.update(fobj=fobj)
    363 evaluation_result_list: List[_LGBM_BoosterEvalMethodResultType] = []
    364 # 检查评估结果

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/basic.py:4143, in Booster.update(self, train_set, fobj)
   4141 if self.__set_objective_to_none:
   4142     raise LightGBMError("无法更新，因为目标函数为null。")
-> 4143 _safe_call(
   4144     _LIB.LGBM_BoosterUpdateOneIter(
   4145         self._handle,
   4146         ctypes.byref(is_finished),
   4147     )
   4148 )
   4149 self.__is_predicted_cur_iter = [False for _ in range(self.__num_dataset)]
   4150 return is_finished.value == 1

File /anaconda/envs/py311/lib/python3.11/site-packages/lightgbm/basic.py:295, in _safe_call(ret)
    287 """检查C API调用的返回值。
    288 
    289 参数
   (...)
    292     C API调用的返回值。
    293 """
    294 if ret != 0:
-> 295     raise LightGBMError(_LIB.LGBM_GetLastError().decode("utf-8"))

LightGBMError: Check failed: (split_indices_block_size_data_partition) > (0) at /home/azureuser/localfiles/LightGBM/lightgbm-python/src/treelearner/cuda/cuda_data_partition.cpp, line 280 ."
}

解决建议 #

1. 检查数据集规模
   该错误大概率由训练样本量过小引发。LightGBM的CUDA实现对最小样本量有要求，样本数太少时无法分配有效CUDA块完成数据分区。

• 确认X_train的样本数量，若小于1000，建议增加样本量，或临时切换到CPU训练。

2. 调整CUDA相关参数
   手动设置CUDA块大小参数，强制分配有效计算单元：

clf = lgb.LGBMClassifier(
    objective='binary',
    device='cuda',
    verbose=1,
    gpu_use_dp=True,
    gpu_device_id=0,
    max_bin=255,
    num_leaves=31
)

3. 验证CUDA环境兼容性

• 确认CUDA版本与LightGBM版本匹配，最新版LightGBM支持CUDA 11.x及以上。
• 检查conda环境中CUDA Toolkit、cuDNN是否安装正常，用nvidia-smi确认GPU被正确识别。

4. 重新编译LightGBM（手动编译场景）
   如果是自行编译的CUDA版本：

• 编译时明确指定CUDA路径：

  cmake -DUSE_CUDA=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda ..
  make -j$(nproc)
  

• 切换到stable分支编译，避免开发分支的潜在bug。

5. 临时切换GPU实现
   若CUDA问题无法快速解决，可改用OpenCL版GPU加速：

clf = lgb.LGBMClassifier(
    objective='binary',
    device='gpu',  # 启用OpenCL加速，兼容性更好
    verbose=1
)

OpenCL对小数据集支持更友好，多数场景下性能与CUDA版本差异不大。

内容的提问来源于stack exchange，提问作者wordsforthewise