Python不用嵌套for循环遍历大文件匹配模型参数的实现方法

方案1：基于pandas实现（最推荐，适配结构化表格场景） #

直接使用pandas的表连接功能完成多字段匹配，全程无需手动写循环，代码简洁且效率极高：

import pandas as pd

# 读取两个文件，若为excel格式可替换为pd.read_excel
df1 = pd.read_csv("文件1路径.csv")
df2 = pd.read_csv("文件2路径.csv")

# 统一匹配字段的列名
df1 = df1.rename(columns={"a1":"a", "b1":"b", "c1":"c", "d1":"d"})
df2 = df2.rename(columns={"a2":"a", "b2":"b", "c2":"c", "d2":"d"})

# 按4个核心参数字段做内连接，直接得到合并后的表
result_df = pd.merge(
    left=df1[["a","b","c","d","age"]],
    right=df2[["a","b","c","d","length"]],
    on=["a","b","c","d"],
    how="inner"
)

# 按需执行age和length的计算，示例为新增比值列
result_df["age_length_ratio"] = result_df["age"] / result_df["length"]

# 导出结果到新文件
result_df.to_csv("合并结果.csv", index=False)

方案2：Python原生实现（无第三方依赖） #

如果不想安装pandas，可以用字典构建哈希映射实现O(1)查找，整体效率和pandas方案接近：

import csv

# 第一步：遍历文件2构建参数到length的映射表
param_to_length = {}
with open("文件2路径.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        # 把4个参数组成元组作为字典的key
        key = (row["a2"], row["b2"], row["c2"], row["d2"])
        # 若length需要数值计算可以提前转float/int
        param_to_length[key] = float(row["length"])

# 第二步：遍历文件1直接查表匹配，生成结果
result_rows = []
with open("文件1路径.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        key = (row["a1"], row["b1"], row["c1"], row["d1"])
        matched_length = param_to_length[key]
        age = float(row["age"])
        # 此处执行你需要的自定义计算
        calculated_val = age * matched_length
        result_rows.append({
            "a": row["a1"],
            "b": row["b1"],
            "c": row["c1"],
            "d": row["d1"],
            "age": age,
            "length": matched_length,
            "custom_calc_result": calculated_val
        })

# 导出结果
with open("合并结果.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=result_rows[0].keys())
    writer.writeheader()
    writer.writerows(result_rows)

方案优势 #

• 时间复杂度从原嵌套循环的O(n²)降至O(n)，16000条数据的场景下运算速度提升上千倍
• 无需手动处理循环索引，避免索引越界、匹配漏判等问题，稳健性更高

内容的提问来源于stack exchange，提问作者Maria