如何基于另一DataFrame移除目标DataFrame序列中的指定字符串

嘿，咱们一步步来解决这个问题！你手头有两个DataFrame，需要根据匹配的ID，把token列表里属于对应低频词的元素删掉。下面分别用PySpark和Pandas给你演示实现方法，看你用哪个工具更顺手～

输入数据示例 #

stringTokenDF #

leastFrequentDf #

PySpark 实现方案 #

首先我们需要将两个DataFrame按Id关联，然后自定义一个UDF（用户自定义函数）来过滤掉Tokens中属于对应低频词的元素。

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf, col
from pyspark.sql.types import ArrayType, StringType

# 初始化SparkSession（如果已经有了可以跳过这步）
spark = SparkSession.builder.appName("FilterLowFreqTokens").getOrCreate()

# 构建示例数据（替换成你自己的数据源即可）
stringToken_data = [
    (1, ["A", "B", "C", "D"]),
    (1, ["B", "C", "D", "G"]),
    (1, ["A", "D", "E"]),
    (1, ["B", "C", "F"]),
    (2, ["A", "C", "D"]),
    (2, ["C", "E", "F"]),
    (2, ["A", "C", "D", "H"])
]
leastFreq_data = [
    (1, ["E", "G"]),
    (2, ["E", "F", "H"])
]

stringTokenDF = spark.createDataFrame(stringToken_data, ["Id", "Tokens"])
leastFrequenctDf = spark.createDataFrame(leastFreq_data, ["Id", "LeastFrequentWords"])

# 定义过滤函数：从tokens列表中移除属于least_freq的元素
def filter_tokens(tokens, least_freq):
    return [token for token in tokens if token not in least_freq]

# 将函数注册为UDF，指定返回类型为字符串数组
filter_udf = udf(filter_tokens, ArrayType(StringType()))

# 关联两个DataFrame，应用UDF得到过滤后的列，最后选择需要的字段
result_df = stringTokenDF.join(leastFrequenctDf, on="Id", how="inner") \
    .withColumn("FilteredTokens", filter_udf(col("Tokens"), col("LeastFrequentWords"))) \
    .select("Id", "Tokens", "FilteredTokens")

# 查看结果
result_df.show(truncate=False)

运行后会得到如下结果：

+---+----------------+----------------+
|Id |Tokens          |FilteredTokens  |
+---+----------------+----------------+
|1  |[A,B,C,D]       |[A,B,C,D]       |
|1  |[B,C,D,G]       |[B,C,D]         |
|1  |[A,D,E]         |[A,D]           |
|1  |[B,C,F]         |[B,C,F]         |
|2  |[A,C,D]         |[A,C,D]         |
|2  |[C,E,F]         |[C]             |
|2  |[A,C,D,H]       |[A,C,D]         |
+---+----------------+----------------+

Pandas 实现方案 #

如果用Pandas处理的话，步骤会更简洁：先通过merge关联两个DataFrame，再用apply结合lambda表达式完成过滤。

import pandas as pd

# 构建示例数据
stringToken_data = {
    "Id": [1,1,1,1,2,2,2],
    "Tokens": [["A","B","C","D"], ["B","C","D","G"], ["A","D","E"], ["B","C","F"], ["A","C","D"], ["C","E","F"], ["A","C","D","H"]]
}
leastFreq_data = {
    "Id": [1,2],
    "LeastFrequentWords": [["E","G"], ["E","F","H"]]
}

stringTokenDF = pd.DataFrame(stringToken_data)
leastFrequenctDf = pd.DataFrame(leastFreq_data)

# 关联数据并过滤tokens列
result_df = stringTokenDF.merge(leastFrequenctDf, on="Id", how="inner")
result_df["FilteredTokens"] = result_df.apply(
    lambda row: [token for token in row["Tokens"] if token not in row["LeastFrequentWords"]],
    axis=1
)

# 展示最终结果
print(result_df[["Id", "Tokens", "FilteredTokens"]])

输出结果如下：

Id           Tokens FilteredTokens
0   1  [A, B, C, D]    [A, B, C, D]
1   1  [B, C, D, G]       [B, C, D]
2   1     [A, D, E]          [A, D]
3   1     [B, C, F]       [B, C, F]
4   2     [A, C, D]    [A, C, D]
5   2     [C, E, F]             [C]
6   2  [A, C, D, H]    [A, C, D]

内容的提问来源于stack exchange，提问作者Mohan