运行PySCENIC grn命令时出现distributed相关警告求助

问题场景 #

我按照论文《A scalable SCENIC workflow for single-cell gene regulatory network analysis》中的流程运行PySCENIC，已使用10x Genomics下载的PBMC数据集生成Loom文件，执行以下命令时出现大量distributed相关警告：

(scenic_protocol) lij@shpc-1392-instance-hgaxepHO:~$ pyscenic grn --num_workers 20 --output adj.tsv --method grnboost2 PBMC10k_filtered.loom hs_hgnc_tfs.txt

错误日志 #

(scenic_protocol) lij@shpc-1392-instance-hgaxepHO:~$ pyscenic grn --num_workers 20 --output adj.tsv --method grnboost2 PBMC10k_filtered.loom hs_hgnc_tfs.txt

2023-06-05 10:02:26,215 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2023-06-05 10:02:30,653 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
/home/lij/miniconda3/envs/scenic_protocol/lib/python3.10/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 39847 instead
warnings.warn(
preparing dask client
parsing input
creating dask graph
2023-06-05 10:09:22,886 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:40481']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:40481']})
2023-06-05 10:09:25,385 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:34745 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:40481',)}
2023-06-05 10:09:48,372 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:44933']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:44933']})
2023-06-05 10:09:50,752 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:41941 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:44933',)}
2023-06-05 10:11:22,756 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:43283', 'tcp://127.0.0.1:33025']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:43283', 'tcp://127.0.0.1:33025']})
2023-06-05 10:11:22,763 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:43283', 'tcp://127.0.0.1:33025']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:43283', 'tcp://127.0.0.1:33025']})
2023-06-05 10:11:25,204 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:35385 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:43283', 'tcp://127.0.0.1:33025')}
2023-06-05 10:11:25,204 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:41555 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:43283', 'tcp://127.0.0.1:33025')}
2023-06-05 10:11:57,216 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:37661', 'tcp://127.0.0.1:41941', 'tcp://127.0.0.1:35717']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:37661', 'tcp://127.0.0.1:41941', 'tcp://127.0.0.1:35717']})
2023-06-05 10:11:57,274 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:37661', 'tcp://127.0.0.1:41941', 'tcp://127.0.0.1:35717']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:37661', 'tcp://127.0.0.1:41941', 'tcp://127.0.0.1:35717']})
2023-06-05 10:11:59,471 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:37643 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:37661', 'tcp://127.0.0.1:41941', 'tcp://127.0.0.1:35717')}
2023-06-05 10:11:59,473 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:39775 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:37661', 'tcp://127.0.0.1:41941', 'tcp://127.0.0.1:35717')}
2023-06-05 10:13:01,197 - distributed.worker - WARNING - Could not find data: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:44933', 'tcp://127.0.0.1:40481', 'tcp://127.0.0.1:34745', 'tcp://127.0.0.1:39775', 'tcp://127.0.0.1:33025']} on workers: [] (who_has: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ['tcp://127.0.0.1:44933', 'tcp://127.0.0.1:40481', 'tcp://127.0.0.1:34745', 'tcp://127.0.0.1:39775', 'tcp://127.0.0.1:33025']})
2023-06-05 10:13:03,690 - distributed.scheduler - WARNING - Worker tcp://127.0.0.1:43555 failed to acquire keys: {'ndarray-bfe6126093e0117e52fb084d3f577fc9': ('tcp://127.0.0.1:44933', 'tcp://127.0.0.1:40481', 'tcp://127.0.0.1:34745', 'tcp://127.0.0.1:39775', 'tcp://127.0.0.1:33025')}

解决方法 #

• 清理残留Dask进程：
  先终止所有关联的Python和Dask进程，避免端口占用和资源残留：

  pkill -f dask
  pkill -f python
  

  注意：如果当前有其他重要Python进程在运行，谨慎使用pkill -f python，可改用ps aux | grep dask找到进程ID后手动杀掉。

• 调整Worker数量：
  命令中指定的--num_workers 20可能远超机器CPU核心数，导致资源竞争和通信故障。建议将worker数调整为与机器核心数匹配（可通过nproc命令查看核心数），比如：

  pyscenic grn --num_workers 8 --output adj.tsv --method grnboost2 PBMC10k_filtered.loom hs_hgnc_tfs.txt
  

• 显式指定Dask端口：
  针对端口8787被占用的警告，通过环境变量指定新的Dask调度器端口：

  export DASK_SCHEDULER_PORT=8788
  pyscenic grn --num_workers 8 --output adj.tsv --method grnboost2 PBMC10k_filtered.loom hs_hgnc_tfs.txt
  

• 禁用分布式模式（备选）：
  如果上述方法无效，可去掉--num_workers参数，让PySCENIC以单进程模式运行，虽然速度较慢，但能彻底避免分布式通信问题：

  pyscenic grn --output adj.tsv --method grnboost2 PBMC10k_filtered.loom hs_hgnc_tfs.txt
  

内容的提问来源于stack exchange，提问作者lij