如何通过任意字符串列匹配正则列表过滤Polars DataFrame？

已知基础操作 #

在Polars中，已有以下两种常用操作：

• 选择所有字符串类型列：

df.select(pl.col(pl.String))

• 针对单个字符串列，用正则表达式过滤行：

df.filter(pl.col("feature").str.contains("dangerous"))

核心需求 #

如何基于正则表达式列表实现DataFrame行过滤：只要某一行的任意字符串列匹配列表中的任意正则模式，就保留该行，否则丢弃？

编辑1：测试用数据 #

以下是用于验证功能与性能的测试DataFrame及正则列表生成代码：

import random
from faker import Faker
import polars as pl

random.seed(42)
Faker.seed(42)

faker = Faker()

df_len = 10000

df = pl.DataFrame(
    [
        pl.Series("a", [random.randint(0, 511) for _ in range(df_len)]).cast(pl.Binary),
        pl.Series("b", [random.randint(0, 1) for _ in range(df_len)]).cast(pl.Boolean),
        pl.Series("c", faker.sentences(df_len), pl.String),
        pl.Series("d", [random.randint(0, 255) for _ in range(df_len)], pl.UInt8),
        pl.Series("e", faker.words(df_len), pl.String),
        pl.Series(
            "f",
            [random.randint(0, 255) * random.TWOPI for _ in range(df_len)],
            pl.Float32,
        ),
        pl.Series("g", faker.words(df_len), pl.String),
    ]
)

patterns = [r"(?i)dangerous", r"always", r"(?i)prevent"]

执行print(df)的输出如下：

shape: (10_000, 7)
┌────────┬───────┬─────────────────────────────────┬─────┬───────────┬─────────────┬──────────┐
│ a      ┆ b     ┆ c                               ┆ d   ┆ e         ┆ f           ┆ g        │
│ ---    ┆ ---   ┆ ---                             ┆ --- ┆ ---       ┆ ---         ┆ ---      │
│ binary ┆ bool  ┆ str                             ┆ u8  ┆ str       ┆ f32         ┆ str      │
╞════════╪═══════╪═════════════════════════════════╪═════╪═══════════╪═════════════╪══════════╡
│ b"114" ┆ false ┆ Agent every development say.    ┆ 164 ┆ let       ┆ 980.17688   ┆ yard     │
│ b"25"  ┆ true  ┆ Beautiful instead ahead despit… ┆ 210 ┆ reach     ┆ 458.672516  ┆ son      │
│ b"281" ┆ false ┆ Information last everything th… ┆ 230 ┆ arm       ┆ 50.265484   ┆ standard │
│ b"250" ┆ false ┆ Choice whatever from behavior … ┆ 29  ┆ operation ┆ 929.911438  ┆ final    │
│ b"228" ┆ false ┆ Page southern role movie win h… ┆ 242 ┆ coach     ┆ 1149.822876 ┆ none     │
│ …      ┆ …     ┆ …                               ┆ …   ┆ …         ┆ …           ┆ …        │
│ b"30"  ┆ true  ┆ Huge course partner.            ┆ 249 ┆ media     ┆ 1118.406982 ┆ movement │
│ b"33"  ┆ true  ┆ Building sign recently avoid u… ┆ 132 ┆ practice  ┆ 282.743347  ┆ big      │
│ b"346" ┆ false ┆ Paper will board.               ┆ 72  ┆ similar   ┆ 376.991119  ┆ just     │
│ b"431" ┆ true  ┆ Technology money worker spring… ┆ 140 ┆ sign      ┆ 94.24778    ┆ audience │
│ b"267" ┆ false ┆ A third traditional ago.        ┆ 40  ┆ available ┆ 615.752136  ┆ always   │
└────────┴───────┴─────────────────────────────────┴─────┴───────────┴─────────────┴──────────┘

编辑2：最优解决方案 #

目前性能最优的实现方式，通过pl.any_horizontal结合字符串列的正则匹配完成过滤：

# 将正则列表合并为单个正则表达式（用|分隔多个规则）
combined_regex = "|".join(patterns)
# 过滤逻辑：任意字符串列匹配正则则保留整行
filtered_df = df.filter(pl.any_horizontal(pl.col(pl.String).str.contains(combined_regex)))

执行上述代码后，过滤结果如下：

shape: (146, 7)
┌────────┬───────┬─────────────────────────────────┬─────┬───────────┬─────────────┬──────────┐
│ a      ┆ b     ┆ c                               ┆ d   ┆ e         ┆ f           ┆ g        │
│ ---    ┆ ---   ┆ ---                             ┆ --- ┆ ---       ┆ ---         ┆ ---      │
│ binary ┆ bool  ┆ str                             ┆ u8  ┆ str       ┆ f32         ┆ str      │
╞════════╪═══════╪═════════════════════════════════╪═════╪═══════════╪═════════════╪══════════╡
│ b"57"  ┆ true  ┆ During prevent accept seem sho… ┆ 137 ┆ various   ┆ 471.238892  ┆ customer │
│ b"269" ┆ true  ┆ Ball always it focus economy b… ┆ 179 ┆ key       ┆ 471.238892  ┆ guy      │
│ b"250" ┆ false ┆ Admit attack energy always.     ┆ 175 ┆ purpose   ┆ 1281.769775 ┆ wonder   │
│ b"82"  ┆ false ┆ Beyond prevent entire staff.    ┆ 242 ┆ hair      ┆ 904.778687  ┆ around   │
│ b"186" ┆ false ┆ Suffer accept letter visit alw… ┆ 134 ┆ magazine  ┆ 12.566371   ┆ dream    │
│ …      ┆ …     ┆ …                               ┆ …   ┆ …         ┆ …           ┆ …        │
│ b"464" ┆ true  ┆ Kid prevent avoid quite brothe… ┆ 153 ┆ visit     ┆ 879.645935  ┆ anything │
│ b"426" ┆ true  ┆ Your sure piece simple always … ┆ 247 ┆ recently  ┆ 1055.575073 ┆ laugh    │
│ b"403" ┆ false ┆ Difference all machine let cha… ┆ 178 ┆ former    ┆ 1061.858276 ┆ always   │
│ b"184" ┆ true  ┆ Morning carry event tell preve… ┆ 3   ┆ entire    ┆ 1432.566284 ┆ hit      │
│ b"267" ┆ false ┆ A third traditional ago.        ┆ 40  ┆ available ┆ 615.752136  ┆ always   │
└────────┴───────┴─────────────────────────────────┴─────┴───────────┴─────────────┴──────────┘

内容的提问来源于stack exchange，提问作者rickhg12hs