Snakemake集群执行时随机出现[Errno 13]权限拒绝错误排查

在SLURM集群运行Snakemake 7.19.1版本的模拟工作流时，会随机触发PermissionError: [Errno 13] Permission denied错误，导致整个工作流中断。目前无法定位错误触发的具体场景，不确定是Snakemake的bug还是使用问题。

错误表现为：Snakemake无法写入某临时.sh脚本，但其他通配符集对应的同类型脚本可正常写入，且运行小型工作流时无此问题。

错误日志 #

Traceback (most recent call last):
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/__init__.py", line 752, in snakemake
    success = workflow.execute(
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/workflow.py", line 1089, in execute
    raise e
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/workflow.py", line 1085, in execute
    success = self.scheduler.schedule()
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/scheduler.py", line 592, in schedule
    self.run(runjobs)
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/scheduler.py", line 641, in run
    executor.run_jobs(
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 155, in run_jobs
    self.run(
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 1156, in run
    self.write_jobscript(job, jobscript)
  File "/home/sjuhel/mambaforge/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 901, in write_jobscript
    with open(jobscript, "w") as f:
PermissionError: [Errno 13] Permission denied: '/data/sjuhel/BoARIO-inputs/.snakemake/tmp.7lly73mm/snakejob.run_generic.11469.sh'

SLURM配置（profile文件） #

cluster:
  mkdir -p /scratchu/sjuhel/logs/smk/{rule} &&
  sbatch
    --parsable
    --mem={resources.mem_mb}
    --job-name=smk-{rule}-{wildcards}
    --cpus-per-task={threads}
    --output=/scratchu/sjuhel/logs/smk/{rule}/{rule}-{wildcards}-%j.out
    --time={resources.time}
    --partition={resources.partition}
default-resources:
  - mem_mb=2000
  - partition=zen16
  - time=60
  - threads=4
restart-times: 0
max-jobs-per-second: 10
max-status-checks-per-second: 1
local-cores: 1
latency-wait: 60
jobs: 16
keep-going: False
rerun-incomplete: True
printshellcmds: True
scheduler: greedy
use-conda: True
conda-frontend: mamba
cluster-status: status-sacct.sh

工作流启动命令 #

nohup snakemake generate_csv_from_all_xp --profile simple > "../Runs/snakelog-$(date +"%FT%H%M%z").log" 2>&1 &

触发错误时的任务统计 #

Using shell: /usr/bin/bash
Provided cluster nodes: 16
Job stats:
job                         count    min threads    max threads
------------------------  -------  -------------  -------------
generate_csv_from_all_xp        1              1              1
generate_csv_from_xp            3              1              1 -> 指标聚合
indicators                   4916              1              1 -> 链接到实验文件夹（多实验可共享结果）
indicators_generic           4799              1              1 -> 模拟生成指标
init_all_sim_parquet            1              1              1
run_generic                  4771              4              4 -> 待运行模拟任务
xp_parquet                      3              1              1 -> 按不同场景分组模拟任务
total                       14494              1              4

排查方向 #

目前怀疑该问题可能与退出SSH会话有关：使用nohup启动的Snakemake进程在断开连接后无法写入文件，计划改用screen尝试，但仍希望明确错误根源。

内容的提问来源于stack exchange，提问作者SJuhel