输出为目录的Checkpoint规则通配符推断错误求助

问题场景 #

运行基于Snakemake的Micro-C测序分析流程时，请求merge_dedup规则输出时触发通配符推断错误，报错指向chunk_fastq规则的lambda输入函数。

报错信息 #

InputFunctionException in line 114 of /tungstenfs/scratch/ggiorget/ilya/seq_data/microC/C-integration-mapping/workflow/Snakefile:
Error:
  KeyError: '1C8+20N+Bxb1_clone_1C1/lane1'
Wildcards:
  library=1C8+20N+Bxb1_clone_1C1/lane1
  run=1.01.fastq.gz
Traceback:
  File "/tungstenfs/scratch/ggiorget/ilya/seq_data/microC/C-integration-mapping/workflow/Snakefile", line 119, in <lambda>

错误对应chunk_fastq规则中fastq1的lambda函数行：LIBRARY_RUN_FASTQS[wildcards.library][wildcards.run], 0。

通过--debug-dag可见，map_chunks_bwa等中间规则的通配符解析正确：

candidate job map_chunks_bwa
    wildcards: library=1C8+20N+Bxb1_clone_1C1, run=lane1, chunk_id=01

且chunk_fastq规则的输出已成功生成。

关键Snakefile代码片段 #

Checkpoint规则 #

checkpoint chunk_fastq:
    input:
        fastq1=lambda wildcards: (
            f"downloaded_fastqs/{wildcards.library}.{wildcards.run}.1.fastq.gz"
            if needs_downloading(
                LIBRARY_RUN_FASTQS[wildcards.library][wildcards.run], 0
            )
            else LIBRARY_RUN_FASTQS[wildcards.library][wildcards.run][0]
        ),
        fastq2=lambda wildcards: (
            f"downloaded_fastqs/{wildcards.library}.{wildcards.run}.2.fastq.gz"
            if needs_downloading(
                LIBRARY_RUN_FASTQS[wildcards.library][wildcards.run], 1
            )
            else LIBRARY_RUN_FASTQS[wildcards.library][wildcards.run][1]
        ),
    params:
        chunksize=lambda wildcards: config["map"]["chunksize"] * 4,
    threads: 4
    output:
        directory("outputs/fastq_chunks/{library}/{run}"),
    shell:
        """
        mkdir -p {output};
        zcat {input.fastq1} | split -l {params.chunksize} -d \
            --filter 'bgzip -c -@ {threads} > $FILE.fastq.gz' - \
            {output}/1.
        zcat {input.fastq2} | split -l {params.chunksize} -d \
            --filter 'bgzip -c -@ {threads} > $FILE.fastq.gz' - \
            {output}/2.
        """

中间处理规则 #

rule map_chunks_bwa:
    input:
        reads=[
            "outputs/fastq_chunks/{library}/{run}/1.{chunk_id}.fastq.gz",
            "outputs/fastq_chunks/{library}/{run}/2.{chunk_id}.fastq.gz",
        ],
        reference=config["genome"]["custom_genome_path"],
        idx=idx,
    params:
        bwa=config["map"]["mapper"],
        extra="-SP",
        sort="none",
        dedup="none",
    log:
        "logs/bwa_memx/{library}/{run}/{chunk_id}.log",
    threads: 8
    output:
        pipe("outputs/mapped_chunks/{library}/{run}/{chunk_id,[0-9]+}.bam"),
    wrapper:
        "v1.24.0/bio/bwa-memx/mem"


rule parse_sort_chunks:
    input:
        bams="outputs/mapped_chunks/{library}/{run}/{chunk_id}.bam",
        chromsizes=config["genome"]["chrom_sizes_path"],
    threads: 8
    params:
        dropsam_flag="--drop-sam" if config["parse"].get("drop_sam", False) else "",
        dropreadid_flag="--drop-readid"
        if config["parse"].get("drop_readid", False)
        else "",
        dropseq_flag="--drop-seq" if config["parse"].get("drop_seq", True) else "",
        parsing_options=config["parse"].get("parsing_options", ""),
        keep_bams_command=f"| tee >(samtools view -bS > outputs/mapped_chunks/{{library}}/{{run}}/{{chunk_id}}.{custom_genome_name}.bam)"
        if config["parse"]["keep_unparsed_bams"]
        else "",
    output:
        f"outputs/parsed_chunks/{{library}}/{{run}}/{{chunk_id,[0-9]+}}.{custom_genome_name}.pairs.gz",
    shell:
        """
        cat {input.bams} {params.keep_bams_command} |
        pairtools parse {params.dropsam_flag} {params.dropreadid_flag} {params.dropseq_flag} \
        {params.parsing_options} \
        -c {input.chromsizes} \
        | pairtools sort --nproc {threads} \
        -o {output} \
        """

聚合及后续规则 #

def get_pair_chunks(wildcards):
    checkpoint_output = checkpoints.chunk_fastq.get(**wildcards).output
    chunk_ids = glob_wildcards(
        f"outputs/fastq_chunks/{wildcards.library}/{wildcards.run}/2.{{chunk_id}}.fastq.gz",
    ).chunk_id
    paths = expand(
        f"outputs/parsed_chunks/{wildcards.library}/{wildcards.run}/{{chunk_id}}.{custom_genome_name}.pairs.gz",
        chunk_id=chunk_ids,
    )
    return paths


rule merge_runs:
    input:
        get_pair_chunks,
    output:
        pipe(f"outputs/parsed_runs/{{library}}/{{run}}.{custom_genome_name}.pairs.gz"),
    shell:
        "echo {wildcards}; pairtools merge {input} --nproc {threads} -o {output}"


rule merge_dedup:
    input:
        pairs=lambda wildcards: expand(
            f"outputs/parsed_runs/{wildcards.library}/{{run}}.{custom_genome_name}.pairs.gz",
            run=list(LIBRARY_RUN_FASTQS[wildcards.library].keys()),
        ),
    params:
        dedup_options=lambda wildcards: config["dedup"].get("dedup_options", ""),
    threads: 8
    output:
        multiext(
            f"outputs/library_pairs/{{library}}.{custom_genome_name}",
            ".nodups.pairs.gz",
            ".nodups.bam",
            ".unmapped.pairs.gz",
            ".unmapped.bam",
            ".dups.pairs.gz",
            ".dups.bam",
            ".dedup.stats",
        ),
    shell:
        f"""pairtools merge {{input.pairs}}) --nproc {{threads}}""" #Some more stuff here

错误原因分析 #

1. 通配符混淆：报错中library通配符被错误解析为1C8+20N+Bxb1_clone_1C1/lane1（包含路径分隔符/），run被解析为1.01.fastq.gz，与中间规则的正确解析结果完全错位。
2. 反向推断逻辑错误：请求merge_dedup输出时，Snakemake需要反向推导上游规则的通配符。由于merge_dedup输出仅包含{library}通配符，其输入通过LIBRARY_RUN_FASTQS动态获取run列表的逻辑，导致Snakemake无法正确区分library和run的边界，将路径中的lane1误判为library的一部分。
3. Checkpoint通配符传递干扰：chunk_fastq作为checkpoint，其输出路径的{library}/{run}结构在正向执行时正常，但反向推断时，路径分隔符导致通配符解析逻辑混乱。

解决办法 #

方案1：添加通配符约束 #

在merge_dedup规则中明确约束library通配符不能包含/，避免路径分隔符干扰解析：

rule merge_dedup:
    wildcards:
        library=regex(r"[^/]+"),  # 限制library不含斜杠
    input:
        pairs=lambda wildcards: expand(
            f"outputs/parsed_runs/{wildcards.library}/{{run}}.{custom_genome_name}.pairs.gz",
            run=list(LIBRARY_RUN_FASTQS[wildcards.library].keys()),
        ),
    # 其余规则内容保持不变

方案2：调整路径命名规范 #

修改所有规则的路径模板，用下划线等非路径分隔符替代/分隔library和run，例如：

• chunk_fastq输出改为：directory("outputs/fastq_chunks/{library}_{run}")
• map_chunks_bwa输入改为："outputs/fastq_chunks/{library}_{run}/1.{chunk_id}.fastq.gz"
  同步修改所有相关规则的路径模板，彻底避免通配符与路径分隔符的冲突。

方案3：重构输入获取逻辑 #

改用glob动态获取merge_dedup的输入，替代直接引用LIBRARY_RUN_FASTQS，让Snakemake能正确推断通配符：

def get_merge_dedup_input(wildcards):
    run_paths = glob_wildcards(
        f"outputs/parsed_runs/{wildcards.library}/{{run}}.{custom_genome_name}.pairs.gz"
    ).run
    return expand(
        f"outputs/parsed_runs/{wildcards.library}/{{run}}.{custom_genome_name}.pairs.gz",
        run=run_paths
    )

rule merge_dedup:
    input:
        pairs=get_merge_dedup_input,
    # 其余规则内容保持不变

方案4：手动指定通配符运行 #

临时测试时，可直接指定library通配符跳过自动推断：

snakemake --cores 8 outputs/library_pairs/1C8+20N+Bxb1_clone_1C1.custom_genome.nodups.pairs.gz

内容的提问来源于stack exchange，提问作者Phlya