为何publishDir无法复制大目录？Nextflow+Slurm环境问题

使用Nextflow的publishDir将Cellranger流程的输出目录（包含子目录）复制到指定路径，小目录复制正常，但14G的大目录无法完成复制。当前环境为Slurm执行器+共享文件系统。

已尝试的解决方案 #

• 清除缓存
• 将OpenJDK替换为Corretto（原使用JetBrains版本）
• 更换其他文件系统
• 尝试同目录下相同大小的其他任务
• 运行小任务（可成功）

Nextflow脚本 #

params.run_dir = ''
params.sample_id = ''
params.csv = ''
params.workflow = ''
params.gex_outdir = '/work/srb108/GEX'


process makefastq_gex {
    cache 'lenient'
    beforeScript 'module load Cell-Ranger/7.2.0'
    publishDir "${params.gex_outdir}", mode: 'copy'

    input:
    path run_dir
    val sample_id
    path csv
    
    output:
    path "${sample_id}_mkfastq_outs"

    script:
    """
    cellranger mkfastq --run=${run_dir} --csv=${csv} --output-dir=${sample_id}_mkfastq_outs --delete-undetermined
    """

} 


workflow {
    run_dir = Channel.fromPath(params.run_dir)
    sample_id = params.sample_id
    csv = Channel.fromPath(params.csv)
    workflow = params.workflow

    if(params.workflow == "gex") {
        makefastq_gex(run_dir, sample_id, csv)
    }  
    
}

配置文件 #

workDir = '/work/srb108'

process {
    executor = 'slurm'

    withName: makefastq_gex {
        cpus = 10
        memory = '250 GB'
        clusterOptions = '--partition=dhvi --job-name=test_mkfastq_job --mail-type=END --mail-user=srb108@duke.edu'
    }

}

成功复制的日志片段 #

Mar-18 14:52:11.105 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=10; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Mar-18 14:52:11.254 [main] DEBUG nextflow.Session - Session await > all processes finished
Mar-18 14:52:11.280 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Mar-18 14:52:11.285 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-18 14:52:16.328 [main] INFO  nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (1 files)
^ THIS LINE REPEATS A FEW TIMES
Mar-18 14:58:42.520 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Mar-18 14:58:43.105 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=7h 41m 42s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=10; peakMemory=100 GB; ]
Mar-18 14:58:43.731 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Mar-18 14:58:44.215 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Mar-18 14:58:44.216 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

失败复制的日志片段 #

Mar-25 16:02:02.696 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]
Mar-25 16:07:02.702 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]
Mar-25 16:12:02.709 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]

注：.command.err文件无任何内容

分析与解决建议 #

从日志对比可见，大任务未触发PublishDir线程池创建，推测核心原因及对应解决方法如下：

1. Slurm任务超时
   大目录复制耗时可能超过Slurm默认超时时间，导致任务被强制终止。检查队列超时设置，或在clusterOptions中显式添加超时参数，例如：

clusterOptions = '--partition=dhvi --job-name=test_mkfastq_job --mail-type=END --mail-user=srb108@duke.edu --time=24:00:00'

2. 共享文件系统IO瓶颈
   14G目录若包含大量小文件，共享文件系统IO性能可能不足以支撑复制，导致Nextflow卡住。可尝试：

• 将publishDir的mode改为link或symlink（允许软链接替代复制时）
• 在配置文件中调整文件传输线程数：

  executor {
      publishDir = {
          threadPoolSize = 20
      }
  }
  

• 用rsync替代默认复制逻辑，在process中手动处理发布：

  process makefastq_gex {
      cache 'lenient'
      beforeScript 'module load Cell-Ranger/7.2.0'
  
      input:
      path run_dir
      val sample_id
      path csv
      
      output:
      path "${sample_id}_mkfastq_outs"
  
      script:
      """
      cellranger mkfastq --run=${run_dir} --csv=${csv} --output-dir=${sample_id}_mkfastq_outs --delete-undetermined
      rsync -avh --progress "${sample_id}_mkfastq_outs" "${params.gex_outdir}/"
      """
  }
  

3. Nextflow主进程内存不足
   大目录复制时，Nextflow需要缓存文件列表等信息，可能超出默认内存限制。在配置文件中增加主进程内存：

executor {
  memory = '16 GB'
}

4. 输出目录异常
   手动进入work目录，检查${sample_id}_mkfastq_outs的结构和文件完整性，确认是否存在特殊字符文件、无限大小文件等异常情况。

内容的提问来源于stack exchange，提问作者Samantha Balgobin