Apple M1芯片设备运行LightGBM报TerminatedWorkerError问题求助

我在使用微软的LightGBM（LGBM）库，当前脚本与能正常运行的XGBoost、随机森林脚本结构高度相似，但在搭载M1芯片的MacBook Pro和MacStudio上运行时，持续抛出以下错误：

joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGSEGV(-11)}

相关代码 #

_train_x, _val_x, _train_y, _val_y = train_test_split(_train_x, _train_y, test_size = 0.2)
    
lgbm_model = LGBMClassifier(bagging_fraction = 0.75, bagging_freq = 5, random_state=42, verbose=-1, force_col_wise=True)
    
kfoldcv = StratifiedKFold(n_splits=3, shuffle=True, random_state=7)
    
lgbm_random_search = RandomizedSearchCV(estimator = lgbm_model, param_distributions = self._param_dict, n_iter = self.num_searches, cv = kfoldcv, verbose=2, random_state=42, n_jobs=-1)
    
lgbm_random_search.fit(_train_x, _train_y)

self._CrossVal_largest_accscore = lgbm_random_search.best_score_
    
lgbm_model = LGBMClassifier(n_jobs=-1, verbose=-1, force_col_wise=True, bagging_fraction = 0.75, bagging_freq = 5, **lgbm_random_search.best_params_)

lgbm_model.fit(_train_x, _train_y, callbacks=[early_stopping(50), log_evaluation(50)], eval_set=[(_val_x,_val_y)])

注意事项 #

移除n_jobs=-1参数后，程序在执行lgbm_random_search.fit(_train_x, _train_y)时直接终止。

环境信息 #

系统软件概览 #

System Version: macOS 14.0 (23A344)
Kernel Version: Darwin 23.0.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled

硬件概览 #

Model Name: MacBook Pro
Model Number: MK1F3B/A
Chip: Apple M1 Pro
Total Number of Cores: 10 (8 performance and 2 efficiency)
Memory: 16 GB
System Firmware Version: 10151.1.1
OS Loader Version: 10151.1.1
Activation Lock Status: Enabled

应用软件 #

Visual Studio Code==1.72.2
python==3.10.12

Python包 #

anaconda-client==1.12.0
anaconda-navigator==2.4.2 
conda==23.7.2
conda-build==3.26.0
joblib==1.3.0
lightgbm==4.0.0
matplotlib==3.7.1
matplotlib-inline==0.1.6
numpy==1.23.5
pandas==2.0.3
scikit-image==0.20.0
scikit-learn==1.3.0
scipy==1.11.1
statsmodels==0.14.0
sympy==1.12
xgboost==2.0.0

已尝试的无效解决方案 #

• 移除n_jobs参数
• 添加pre_dispatch=2参数
• 重装相关依赖库
• 调整estimators数量

内容的提问来源于stack exchange，提问作者JonT