PySpark DataFrame多字符串同时包含筛选：rlike失效问题求助

一、rlike失效原因 #

1. 变量名笔误：代码中拼接正则时用了未定义的web_field，实际应该用定义好的wordList，导致正则表达式生成错误。
2. 正则逻辑错误：正则语法里没有&作为“同时满足”的运算符，直接用&拼接单词会生成one&two&three这类无效正则，无法匹配任何符合要求的字符串。

二、简洁解决方案 #

方案1：正则正向预查实现（推荐） #

利用正则的正向预查语法(?=.*{单词})，可以实现“字符串同时包含多个子串”的逻辑，每个单词对应一个预查规则，拼接后作为rlike的参数：

from pyspark.sql import SparkSession

# 初始化SparkSession示例
spark = SparkSession.builder.appName("multi_word_filter").getOrCreate()

# 构建示例DataFrame
data = [
    ("they are one, two, three, four, five, six", "typeA", "objectA"),
    ("they are one, two", "typeB", "objectB"),
    ("they are four,five", "typeC", "objectC"),
    ("they are six, five, four, three, two, one", "typeD", "objectD"),
    ("they are six, one, five, three, two, four", "typeE", "objectE")
]
df = spark.createDataFrame(data, ["message", "type", "object"])

wordList = ["one", "two", "three", "four", "five", "six"]
# 构建正向预查正则：每个规则要求字符串包含对应单词
regex_pattern = "".join([f"(?=.*{word})" for word in wordList])
# 执行筛选
result_df = df.filter(df.message.rlike(regex_pattern))
result_df.show(truncate=False)

方案2：reduce组合contains条件 #

如果对正则不熟悉，可以用functools.reduce将多个contains条件按AND逻辑组合，避免重复编写条件：

from pyspark.sql import functions as F
from functools import reduce

# 生成每个单词的contains判断条件
conditions = [F.col("message").contains(word) for word in wordList]
# 用reduce将所有条件按AND逻辑拼接
result_df = df.filter(reduce(lambda a, b: a & b, conditions))
result_df.show(truncate=False)

两种方案都能输出预期结果，方案1适合大量条件的场景，方案2更直观易读。

内容的提问来源于stack exchange，提问作者peace