Nextflow流程跳过DBcreation进程致下游MM进程失败求助

问题描述 #

基于Nextflow和GATK搭建的分析流程里，panelnormalsone进程之前的所有步骤均正常执行，但流程会直接跳过DBcreation进程，转而运行tumorsamplenamerun进程，最终导致依赖DBcreation输出的MM进程执行失败。单独运行到DBcreation进程时可正常执行，但全流程运行时该进程会被跳过。

相关代码片段 #

process normalsamplenamerun {
  
  publishDir params.trim, mode : 'copy' 

  input:
  tuple val(pair_id), file(recal_readsbamfile)

  output:
  tuple val(pair_id),file(normal_samplefile), emit : SM_normalbam
  tuple val(pair_id),file(normal_samplefile_bai), emit : SM_normal_bai


  script:
  normal_samplefile = pair_id + '_recal_reads_SM.bam'
  normal_samplefile_bai = pair_id + '_recal_reads_SM.bai'

  """
    
  ./gatk --java-options "-Xmx24G" AddOrReplaceReadGroups --INPUT $recal_readsbamfile    --OUTPUT $normal_samplefile --RGLB lib1 --RGPL illumina --RGPU NONE --RGSM NORMAL_$pair_id
  ./gatk --java-options "-Xmx24G" BuildBamIndex --INPUT $normal_samplefile
  """

}

process panelnormalsone {

publishDir params.trim, mode : 'copy'

input:
tuple val(pair_id), file(normal_samplefile)
tuple val(pair_id), file(normal_samplefile_bai)

output:
tuple val(pair_id),file(panelvcfnormal_samplefile) emit : vcf_normal
tuple val(pair_id),file(panelvcfnormal_samplefile_stats) emit : vcf_stats
tuple val(pair_id),file(panelvcfnormal_samplefile_tbi)  emit : vcf_tbi


script:
panelvcfnormal_samplefile = pair_id + '_recal_somatic.vcf.gz'
panelvcfnormal_samplefile_stats = pair_id + '_recal_somatic.vcf.gz.stats'
panelvcfnormal_samplefile_tbi  =   pair_id + '_recal_somatic.vcf.gz.tbi'

"""

./gatk --java-options "-Xmx24G" Mutect2 -R ${params.reference} -I $normal_samplefile --max-mnp-distance 0 -O $panelvcfnormal_samplefile
"""

}

process DBcreation {

publishDir params.trim, mode : 'copy'

input:
file(panelvcfnormal_samplefile)
file(panelvcfnormal_samplefile_stats)
file(panelvcfnormal_samplefile_tbi)

output:
//path "${params.pondb}"
   
script:
sampleOpts = panelvcfnormal_samplefile.collect { p -> ("-V ${p.name}") } .join(' ')

""" 

./gatk --java-options "-Xmx24G" GenomicsDBImport -R ${params.reference} -L  ${params.intervalfile} --genomicsdb-workspace-path ${params.pondb} ${sampleOpts} --merge-input-intervals true --reader-threads 10 
./gatk --java-options "-Xmx24G" CreateSomaticPanelOfNormals -R ${params.reference} --germline-resource ${params.germlineresource} -V gendb://${params.pondb} -O ${params.ponsfile}
"""
}

process tumorsamplenamerun {
  
  publishDir params.trim, mode : 'copy' 

  input:
  tuple val(pair_id), file(recal_readsbamfile)

  output:
  tuple val(pair_id),file(tumor_samplefile), emit : SM_tumorbam
  tuple val(pair_id),file(tumor_samplefile_bai), emit : SM_tumorbam_bai


  script:
  tumor_samplefile = pair_id + '_recal_reads_SM.bam'
  tumor_samplefile_bai = pair_id + '_recal_reads_SM.bai'

  """
    
  ./gatk --java-options "-Xmx24G" AddOrReplaceReadGroups --INPUT $recal_readsbamfile --OUTPUT $tumor_samplefile --RGLB lib1 --RGPL illumina --RGPU NONE --RGSM TUMOR_$pair_id
  ./gatk --java-options "-Xmx24G" BuildBamIndex --INPUT $tumor_samplefile
  """
}

workflow {

  panelnormalsone(normalsamplenamerun.out)
  samples = panelnormalsone.out.vcf_normal | map {  t -> t[1] }  
  stats = panelnormalsone.out.vcf_stats | map {  t -> t[1] } 
  tbi = panelnormalsone.out.vcf_tbi | map {  t -> t[1] } 
  DBcreation(samples.collect(),stats.collect(),tbi.collect()) 
  gatkrun.out.recal_readsbamfile
  | filter { pair_id, recal_readsbamfile -> !pair_id.contains("N") }
  | tumorsamplenamerun
}

（注：已修正脚本中./gatk./gatk的笔误为./gatk）

运行日志 #

[97/f6890e] process > trimming (trim on {CaGB17_T}) [100%] 10 of 10 ✔
 [c7/6f21fd] process > BWAMEM (10)                   [100%] 10 of 10 ✔
 [ed/20534d] process > gatkrun (10)                  [100%] 6 of 6
 [77/5513b3] process > normalsamplenamerun (5)       [100%] 5 of 5
 [80/c3035e] process > panelnormalsone (5)           [100%] 5 of 5
 [-        ] process > DBcreation            -
 [6a/9657cf] process > tumorsamplenamerun (1)        [100%] 1 of 1
 [d3/5b9f35] process > MM (1)                        [100%] 1 of 1, failed: 1

问题原因与修复方案 #

核心原因 #

1. 输入通道处理错误：使用.collect()将Nextflow通道转换为静态列表，而Nextflow进程只能接收通道作为输入，无法识别静态列表作为依赖，导致DBcreation被判定为无输入依赖，直接跳过。
2. 输出未定义：DBcreation的输出被注释，Nextflow无法追踪其产物，也无法识别下游进程对它的依赖，进一步导致调度时忽略该进程。

修复步骤 #

1. 修复DBcreation的输入通道传递

将三个分散的通道合并为一个元组通道，去掉.collect()，保留通道类型传递给进程：

workflow {
  panelnormalsone(normalsamplenamerun.out)
  // 合并三个输出通道为一个包含所有文件的元组通道
  db_input = panelnormalsone.out.vcf_normal
             | combine(panelnormalsone.out.vcf_stats, by: 0)
             | combine(panelnormalsone.out.vcf_tbi, by: 0)
             | map { pair_id, vcf, stats, tbi -> [vcf, stats, tbi] }
  DBcreation(db_input)

  gatkrun.out.recal_readsbamfile
  | filter { pair_id, recal_readsbamfile -> !pair_id.contains("N") }
  | tumorsamplenamerun
}

同时修改DBcreation的输入定义为元组接收：

process DBcreation {
  // ... 其他配置不变
  input:
  tuple file(panelvcfnormal_samplefile), file(panelvcfnormal_samplefile_stats), file(panelvcfnormal_samplefile_tbi)
  // ... 其他配置不变
}

2. 正确定义DBcreation的输出

取消注释输出，并明确指定产物路径，确保Nextflow能追踪输出，同时让下游MM进程可以依赖：

process DBcreation {
  publishDir params.trim, mode : 'copy'

  input:
  tuple file(panelvcfnormal_samplefile), file(panelvcfnormal_samplefile_stats), file(panelvcfnormal_samplefile_tbi)

  output:
  path "${params.pondb}/**", emit: pondb_dir
  path "${params.ponsfile}", emit: pons_file

  script:
  sampleOpts = panelvcfnormal_samplefile.collect { p -> ("-V ${p.name}") } .join(' ')

  """ 
  ./gatk --java-options "-Xmx24G" GenomicsDBImport -R ${params.reference} -L ${params.intervalfile} --genomicsdb-workspace-path ${params.pondb} ${sampleOpts} --merge-input-intervals true --reader-threads 10 
  ./gatk --java-options "-Xmx24G" CreateSomaticPanelOfNormals -R ${params.reference} --germline-resource ${params.germlineresource} -V gendb://${params.pondb} -O ${params.ponsfile}
  """
}

3. 显式关联MM进程与DBcreation的输出

在workflow中确保MM进程接收DBcreation的输出，示例：

MM(tumorsamplenamerun.out, DBcreation.out.pons_file)

验证 #

修改完成后重新运行流程，检查DBcreation是否被正常调度执行，同时确认MM进程能获取到所需的输入文件并成功运行。

内容的提问来源于stack exchange，提问作者sawasthi