基于Snakemake的PRINCESS软件在SLURM集群报错找不到临时文件，登录节点运行正常

问题基本说明 #

• 计划在所属Orion计算集群上，通过SLURM作业调度运行基于Snakemake框架开发的开源分析软件PRINCESS
• 测试验证结果：软件在登录节点可正常完成计算流程；但通过SLURM提交调度作业运行时，会报错snakemake/temp file is not found；若不使用集群调度、本地串行运行，计算耗时远超可接受范围，需定位问题根因
• 已开展的排查操作（均未解决问题）：
  • 与集群管理员核对Snakemake对应的SLURM配置YAML文件，确认配置项与集群管理规则完全匹配，无参数错配
  • 为运行命令添加--latency-wait 120参数，作业运行状态无明显改善
  • 与集群运维团队确认，不存在计算节点限制临时文件读取、删除的访问规则

实际执行的运行命令 #

# 加载所需分析依赖模块
module load PyYAML/5.1.2-GCCcore-8.3.0
module load Miniconda3/4.7.10
module load snakemake/5.30.1
module load HTSlib 

# 模块加载过程中自动触发的关联版本变更
# 1) GCCcore/8.3.0 => GCCcore/9.3.0
# 2) XZ/5.2.4-GCCcore-8.3.0 => XZ/5.2.5-GCCcore-9.3.0
# 3) binutils/2.32-GCCcore-8.3.0 => binutils/2.34-GCCcore-9.3.0
# 4) bzip2/1.0.8-GCCcore-8.3.0 => bzip2/1.0.8-GCCcore-9.3.0
# 5) zlib/1.2.11-GCCcore-8.3.0 => zlib/1.2.11-GCCcore-9.3.0

# 提交PRINCESS分析任务的实际运行命令
python3.7  /net/fs-1/Transpose/Software/princess/princess all -d $PWD/0406/2.without.e -r ont -s $PWD/test.fastq.gz --chr ssa18 -f /mnt/SCRATCH/kristenl/Reference/Simon_Sept2020.fasta -j 200  --rerun-incomplete --verbose 

核心报错信息 #

[Tue Apr  6 09:05:56 2021]
Error in rule readsStat:
    jobid: 6
    output: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/statitics/raw_reads/reads_stat.txt
    conda-env: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/.snakemake/conda/1b58da75
    shell:
        
        python scripts/rawcoverage.py -i /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/test.fastq.gz -o /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/statitics/raw_reads/reads_stat.txt -t 5
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: 12727829

Error executing rule readsStat on cluster (jobid: 6, external: 12727829, jobscript: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/.snakemake/tmp.4xwny_7b/snakejob.readsStat.6.sh). For error details see the cluster log and the log files of the involved rule(s).

各节点作业输出日志摘要 #

• SLURM作业ID 12727829 输出日志：

  Waiting at most 5 seconds for missing files.
  Missing files after 5 seconds:
  /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/.snakemake/tmp.4xwny_7b
  

• SLURM作业ID 12727823 输出日志：

  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 5
  Rules claiming more threads will be scaled down.
  Job counts:
      count   jobs
      1   readsStat
      1
  Select jobs to execute...
  
  [Tue Apr  6 09:05:21 2021]
  Job 0: Calculating read coverage statitics for: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/test.fastq.gz
  
  
          python scripts/rawcoverage.py -i /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/test.fastq.gz -o /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/statitics/raw_reads/reads_stat.txt -t 5
          
  Activating conda environment: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/.snakemake/conda/1b58da75
  Waiting at most 5 seconds for missing files.
  MissingOutputException in line 10 of /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/modules/stat.smk:
  Job Missing files after 5 seconds:
  /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/statitics/raw_reads/reads_stat.txt
  This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
  Job id: 0 completed successfully, but some output files are missing. 0
    File "/cluster/software/snakemake/5.30.1/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 575, in handle_job_success
    File "/cluster/software/snakemake/executors/__init__.py", line 254, in handle_job_success
  Shutting down, this might take some time.
  Exiting because a job execution failed. Look above for error message
  

• SLURM作业ID 12727825（登录节点成功运行的对照日志）：

  Building DAG of jobs...
  Using shell: /usr/bin/bash
  Provided cores: 1 (use --cores to define parallelism)
  Rules claiming more threads will be scaled down.
  Job counts:
      count   jobs
      1   readsStat
      1
  Select jobs to execute...
  
  [Tue Apr  6 09:05:46 2021]
  Job 0: Calculating read coverage statitics for: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/test.fastq.gz
  
  
          python scripts/rawcoverage.py -i /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/test.fastq.gz -o /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/statitics/raw_reads/reads_stat.txt -t 1
          
  Activating conda environment: /net/cn-1/mnt/SCRATCH/princesstest/0406/2.without.e/.snakemake/conda/1b58da75
  [Tue Apr  6 09:06:02 2021]
  Finished job 0.
  1 of 1 steps (100%) done
  

内容的提问来源于stack exchange，提问作者Marie