Snakemake规则无法识别其他规则输出文件的问题排查

问题场景 #

编写了prepare_tuples.smk作为主Snakefile的一部分，其中prepare_tuples设定为主规则，其输入定义为hadd_tuples规则的输出。执行命令：

snakemake prepare_tuples -F -c20

时出现缺失输入文件的错误，Snakemake未自动识别并执行hadd_tuples及上游的cut、truthmatch规则。

错误信息 #

Missing input files for rule prepare_tuples:
    affected files:
        /ceph/users/jmainusch/data/Bs2MuMu/2024_data_stripped.root
        /ceph/users/jmainusch/simulation/Bs2MuMu/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/simulation/Bs2JpsiPhi/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/simulation/Bu2JpsiK/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/data/Bd2MuMu/2024_data_stripped.root
        /ceph/users/jmainusch/data/Bs2JpsiPhi/2024_data_stripped.root
        /ceph/users/jmainusch/data/Bs2KK/2024_data_stripped.root
        /ceph/users/jmainusch/simulation/Bd2MuMu/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/simulation/Bd2KPi/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/data/Bd2KPi/2024_data_stripped.root
        /ceph/users/jmainusch/simulation/Bd2PiPi/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/data/Bu2JpsiK/2024_data_stripped.root
        /ceph/users/jmainusch/simulation/Bs2KK/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/data/Bd2PiPi/2024_data_stripped.root
        /ceph/users/jmainusch/simulation/Bs2KPi/2024_data_stripped_truth-matched.root
        /ceph/users/jmainusch/data/Bs2KPi/2024_data_stripped.root

关联的Snakefile代码 #

configfile: "config/config.yaml"

path = config["repopath"]
inpath = config["cephpath"]
outpath = config["userpath"]
dataset = config["dataset"]

configfile: path + "/1_prepare_tuples/configs/tuples.yaml"

wildcard_constraints:
    src = "data|simulation",
    tuple = "^B.*"

rule prepare_tuples:
    input:
        data = expand(outpath + "data/{tuple}/2024_data_stripped.root", tuple = config["tuples"].keys()),
        sim = expand(outpath + "simulation/{tuple}/2024_data_stripped_truth-matched.root", tuple = config["tuples"].keys()),

rule cut:
    input: 
        script = path + "1_prepare_tuples/scripts/cut.py",
        in_data = inpath + "{src}/{tuple}/2024/{magnet}/data.root",
        cuts = path + "1_prepare_tuples/configs/cuts.yaml",
        truth_vars = path + "1_prepare_tuples/configs/truth-match.yaml",
        control_vars = path + "3_control_plots/configs/channels/{tuple}/plots.yaml",
        BDTS_vars = path + "4_BDTS/configs/variables.yaml",
    params:
        eff_path = "results/efficiencies.yaml",
        tuple_config = path + "/1_prepare_tuples/configs/tuples.yaml",
    output: 
        out_data = outpath + "{src}/{tuple}/{magnet}/2024_data_stripped.root",
    shell: 
        """
        python {input.script} \
        --path {input.in_data} \
        --channel {wildcards.tuple} \
        --tuple_config {params.tuple_config} \
        --source {wildcards.src} \
        --magnet {wildcards.magnet} \
        --outpath {output.out_data} \
        --effpath {params.eff_path} \
        --cuts {input.cuts} \
        --truth_vars {input.truth_vars} \
        --control_vars {input.control_vars} \
        --BDTS_vars {input.BDTS_vars} \
        """
        
rule truthmatch:
    input: 
        script = path + "1_prepare_tuples/scripts/truthmatch.py",
        in_data = outpath + "simulation/{tuple}/{magnet}/2024_data_stripped.root",
        cuts = path + "1_prepare_tuples/configs/truth-match.yaml",
        cut_vars = path + "1_prepare_tuples/configs/cuts.yaml",
        control_vars = path + "3_control_plots/configs/channels/{tuple}/plots.yaml",
        BDTS_vars = path + "4_BDTS/configs/variables.yaml",
    params:
        eff_path = "results/efficiencies.yaml",
        tuple_config = path + "/1_prepare_tuples/configs/tuples.yaml",
    output: 
        out_data = outpath + "simulation/{tuple}/{magnet}/2024_data_stripped_truth-matched.root",
    shell: 
        """
        python {input.script} \
        --path {input.in_data} \
        --channel {wildcards.tuple} \
        --tuple_config {params.tuple_config} \
        --magnet {wildcards.magnet} \
        --outpath {output.out_data} \
        --effpath {params.eff_path} \
        --cuts {input.cuts} \
        --cut_vars {input.cut_vars} \
        --control_vars {input.control_vars} \
        --BDTS_vars {input.BDTS_vars} \
        """

rule hadd_tuples:
    input:
        up = outpath + "{src}/{tuple}/MagUp/2024_data_{mod}.root",
        down = outpath + "{src}/{tuple}/MagDown/2024_data_{mod}.root"
    output:
        outpath + "{src}/{tuple}/2024_data_{mod}.root",
    shell:
        "hadd {output} {input.up} {input.down}"

问题根源 #

1. 通配符不匹配：prepare_tuples的输入文件名（如2024_data_stripped.root、2024_data_stripped_truth-matched.root）与hadd_tuples的输出模式2024_data_{mod}.root没有建立关联。Snakemake无法识别stripped和stripped_truth-matched是mod通配符的取值，因此无法触发hadd_tuples规则。
2. 依赖链断裂：对于simulation类型的文件，hadd_tuples需要的MagUp/MagDown文件由truthmatch生成，但当前规则定义中没有通过通配符将两者关联起来；data类型文件同理，hadd_tuples与上游cut规则的关联也未正确建立。

解决方案 #

1. 修改prepare_tuples规则的输入定义 #

将输入文件名中的固定后缀替换为通配符{mod}，并通过expand指定mod的取值，让Snakemake能匹配到hadd_tuples的输出模式：

rule prepare_tuples:
    input:
        data = expand(outpath + "data/{tuple}/2024_data_{mod}.root", tuple=config["tuples"].keys(), mod="stripped"),
        sim = expand(outpath + "simulation/{tuple}/2024_data_{mod}.root", tuple=config["tuples"].keys(), mod="stripped_truth-matched"),

2. 验证依赖链 #

执行dry-run命令检查Snakemake是否能正确识别所有上游规则：

snakemake prepare_tuples -n -c20

如果输出中包含cut、truthmatch、hadd_tuples的执行计划，说明依赖链已正确建立。

3. 重新执行主规则 #

确认dry-run无问题后，重新运行原命令即可：

snakemake prepare_tuples -F -c20

内容的提问来源于stack exchange，提问作者jmai