如何对比两个Polars DataFrame并填充布尔值：校验模式匹配

基于列A关联两个Polars DataFrame，为df1新增VALID布尔列，标记规则如下：

• 检查df1的C列（存储为字典格式字符串）的完整字典内容是否存在于df2对应A分组的valid_patterns列中；
• 或者，C列字典的任意键是否存在于df2对应A分组的valid_patterns列中；
  满足任一条件则标记为True，否则为False。

输入df1 #

df1 = pl.DataFrame({
   "A": ["a", "b", "a", "c","a","c","a","b","b"], 
   "B": [23,45,60,50,56,44,34,30,97], 
   "C": ["{'pattern1':['a','score1'],'pattern2':['df','text1']}",
        "{'pattern3':['abc','score1']}",
        "{'pattern1':['a','score2'],'pattern4':['df','text2']}",
        "{'pattern5':['a','score2'],'pattern6':['gh','text1'],'pattern7':['gh','text1']}",
        "{'pattern8':['a','score1']}",
        "{'pattern9':['a','score1'],'pattern10':['df','text1']}",
        "{'pattern11':['a','score1']}",
        "{'pattern12':['a','score1'],'pattern13':['df','text1']}",
        "{'pattern3':['a','score1'],'pattern5':['df','text1']}"]
})

对应的DataFrame结构：

shape: (9, 3)
┌─────┬─────┬─────────────────────────────────────────────────────────────────────────────────┐
│ A   ┆ B   ┆ C                                                                               │
│ --- ┆ --- ┆ ---                                                                             │
│ str ┆ i64 ┆ str                                                                             │
╞═════╪═════╪═════════════════════════════════════════════════════════════════════════════════╡
│ a   ┆ 23  ┆ {'pattern1':['a','score1'],'pattern2':['df','text1']}                           │
│ b   ┆ 45  ┆ {'pattern3':['abc','score1']}                                                   │
│ a   ┆ 60  ┆ {'pattern1':['a','score2'],'pattern4':['df','text2']}                           │
│ c   ┆ 50  ┆ {'pattern5':['a','score2'],'pattern6':['gh','text1'],'pattern7':['gh','text1']} │
│ a   ┆ 56  ┆ {'pattern8':['a','score1']}                                                     │
│ c   ┆ 44  ┆ {'pattern9':['a','score1'],'pattern10':['df','text1']}                          │
│ a   ┆ 34  ┆ {'pattern11':['a','score1']}                                                    │
│ b   ┆ 30  ┆ {'pattern12':['a','score1'],'pattern13':['df','text1']}                         │
│ b   ┆ 97  ┆ {'pattern3':['a','score1'],'pattern5':['df','text1']}                           │
└─────┴─────┴─────────────────────────────────────────────────────────────────────────────────┘

输入df2 #

df2 = pl.DataFrame({
   "A": ["a", "a", "b", "b","a","c"], 
   "valid_patterns": ['pattern1',
       "{'pattern1': ['a', 'score2'], 'pattern4': ['df', 'text2']}",
        "{'pattern3':['abc','score1']}",
        'pattern3',
        "{'pattern11': ['a', 'score1']}",
        'pattern9']
})

对应的DataFrame结构：

shape: (6, 2)
┌─────┬────────────────────────────────────────────────────────────┐
│ A   ┆ valid_patterns                                             │
│ --- ┆ ---                                                        │
│ str ┆ str                                                        │
╞═════╪════════════════════════════════════════════════════════════╡
│ a   ┆ pattern1                                                   │
│ a   ┆ {'pattern1': ['a', 'score2'], 'pattern4': ['df', 'text2']} │
│ b   ┆ {'pattern3':['abc','score1']}                              │
│ b   ┆ pattern3                                                   │
│ a   ┆ {'pattern11': ['a', 'score1']}                             │
│ c   ┆ pattern9                                                   │
└─────┴────────────────────────────────────────────────────────────┘

输出示例 #

output = pl.DataFrame({
   "A": ["a", "b", "a", "c","a","c","a","b","b"], 
   "B": [23,45,60,50,56,44,34,30,97], 
   "C": ["{'pattern1':['a','score1'],'pattern2':['df','text1']}",
        "{'pattern3':['abc','score1']}",
        "{'pattern1':['a','score2'],'pattern4':['df','text2']}",
        "{'pattern5':['a','score2'],'pattern6':['gh','text1'],'pattern7':['gh','text1']}",
        "{'pattern8':['a','score1']}",
        "{'pattern9':['a','score1'],'pattern10':['df','text1']}",
        "{'pattern11':['a','score1']}",
        "{'pattern12':['a','score1'],'pattern13':['df','text1']}",
        "{'pattern3':['a','score1'],'pattern5':['df','text1']}"],
    "VALID": [True,True,True,False,False,True,True,False,False]
})

实现方案 #

步骤说明 #

1. 解析字典字符串：将df1.C和df2.valid_patterns中的字典格式字符串转换为实际字典，同时提取df1.C字典的键集合；
2. 分组预处理df2：按A列分组，整理每个分组下的valid_patterns，区分普通键和完整字典，分别存储为集合以提升匹配效率；
3. 关联匹配：将预处理后的规则关联到df1，检查每行是否满足匹配条件，生成VALID列。

代码实现 #

import polars as pl
import ast

# 解析函数：尝试将字符串转为字典，失败则返回原字符串（处理普通键场景）
def parse_pattern(s):
    try:
        return ast.literal_eval(s)
    except (SyntaxError, ValueError):
        return s

# 处理df1：解析C列字典，提取键集合
df1_processed = df1.with_columns(
    pl.col("C").map_elements(parse_pattern).alias("C_dict"),
    pl.col("C").map_elements(lambda x: set(ast.literal_eval(x).keys()) if isinstance(ast.literal_eval(x), dict) else set()).alias("C_keys")
)

# 处理df2：按A分组，整理有效键和有效字典集合
df2_grouped = df2.with_columns(
    pl.col("valid_patterns").map_elements(parse_pattern).alias("parsed_pattern")
).group_by("A").agg(
    pl.col("parsed_pattern").filter(~pl.col("parsed_pattern").is_struct()).alias("valid_keys"),
    pl.col("parsed_pattern").filter(pl.col("parsed_pattern").is_struct()).alias("valid_dicts")
).with_columns(
    pl.col("valid_keys").list.to_set().alias("valid_keys_set"),
    pl.col("valid_dicts").list.to_set().alias("valid_dicts_set")
)

# 关联并生成VALID列
result = df1_processed.join(df2_grouped, on="A", how="left").with_columns(
    (pl.col("C_dict").is_in(pl.col("valid_dicts_set")) | 
     pl.col("C_keys").list.set_intersection(pl.col("valid_keys_set")).list.len() > 0).alias("VALID")
).select(df1.columns + ["VALID"])

print(result)

代码解释 #

• parse_pattern：兼容两种输入格式，自动识别并解析字典字符串；
• df1处理：生成C_dict存储解析后的完整字典，C_keys存储字典的键集合，为后续匹配做准备；
• df2分组处理：按A聚合后，分别收集普通键和完整字典并转为集合，集合操作能大幅提升匹配速度；
• 关联匹配：通过左连接将规则映射到df1，用布尔逻辑判断两个匹配条件，最终保留原df1列和新增的VALID列。

内容的提问来源于stack exchange，提问作者Suresh Subramaniyan