如何基于配置文件循环递进过滤Pandas DataFrame？

问题场景 #

示例数据集：

import pandas as pd

data = {"Subject":["1","2","3","3","4","5","5"],
    "date": ["2020-05-01 16:54:25","2020-05-03 10:31:18","2020-05-08 10:10:40","2020-05-08 10:10:42","2020-05-06 09:30:40","2020-05-07 12:46:30","2020-05-07 12:55:10"],
    "Accept": ["True","False","True","True","False","True","True"],
    "Amount" : [150,30,32,32,300,100,50],
    "accept_1": ["True","False","True","True","False","True","True"],
    "amount_1" : [20,30,32,32,150,100,30],
    "Transaction":["True","True","False","False","True","True","False"],
    "Label":["True","True","True","False","True","True","True"]}
data = pd.DataFrame(data)

示例配置文件：

config = [{"colname": "Accept","KeepValue":"True","RemoveTrues":"True"},
    {"colname":"Transaction","KeepValue":"False","RemoveTrues":"False"}]

需求是递进式过滤：先对原始数据应用第一个规则，后续规则基于上一次过滤后的结果执行，但原代码每次都基于原始data过滤，导致无法实现递进。

原问题代码：

for i in range(len(config)):
    filtering = config[i]
    if filtering["RemoveTrues"] == "True":
        col = filtering["colname"]
        test  = data[data[col] == filtering["KeepValue"]]
        print(test)
    else:
        col = filtering["colname"]
        test = data[(data[col]== filtering["KeepValue"]) | data["Label"]]
        print(test)

问题原因 #

原代码每次过滤都直接使用原始的data变量，没有将每次过滤后的结果保存为新的数据源，导致后续过滤始终从初始数据集开始，无法实现递进式的叠加过滤。

修改后的代码 #

核心思路是维护一个迭代的过滤结果变量，每次过滤都基于这个变量，过滤完成后更新它：

# 初始化过滤结果为原始数据的副本，避免修改原数据
filtered_data = data.copy()

for idx, filtering in enumerate(config):
    col = filtering["colname"]
    keep_val = filtering["KeepValue"]
    
    if filtering["RemoveTrues"] == "True":
        # 只保留指定列等于KeepValue的行
        filtered_data = filtered_data[filtered_data[col] == keep_val]
    else:
        # 保留指定列等于KeepValue 或者 Label为True的行
        filtered_data = filtered_data[(filtered_data[col] == keep_val) | filtered_data["Label"]]
    
    # 打印当前过滤后的结果
    print(f"第{idx+1}次过滤结果：")
    print(filtered_data)
    print("-"*50)

代码说明 #

1. 用filtered_data = data.copy()初始化过滤结果，避免操作过程中修改原始数据集。
2. 使用enumerate遍历配置，方便标记过滤次数。
3. 每次过滤都基于filtered_data操作，并且将过滤结果重新赋值给filtered_data，确保下一次过滤使用的是上一次的结果。
4. 添加了清晰的过滤次数标识，便于查看每一步的输出变化。

内容的提问来源于stack exchange，提问作者Almosino