如何高效筛选DataFrame中关键词存在于描述列的行？

问题场景 #

给定如下DataFrame：

Description     keyword
1  plays the piano   plays
2  plays the piano   write
3  plays the piano   piano
4 knows how to write  the
5 knows how to write  to

需要筛选出每行keyword列值存在于对应Description列值中的行，预期结果：

Description     keyword
1  plays the piano   plays
3  plays the piano   piano
5 knows how to write  to

高效实现方法 #

下面按效率从高到低推荐几种方案：

1. 列表推导式（优先推荐）

纯Python循环的列表推导式在处理DataFrame逐行判断时，效率远高于apply，代码也简洁：

import pandas as pd

# 构造示例DataFrame
df = pd.DataFrame(
    {
        'Description': ['plays the piano', 'plays the piano', 'plays the piano', 'knows how to write', 'knows how to write'],
        'keyword': ['plays', 'write', 'piano', 'the', 'to']
    },
    index=[1,2,3,4,5]
)

# 生成筛选掩码
mask = [kw in desc for desc, kw in zip(df['Description'], df['keyword'])]
# 应用筛选
filtered_df = df[mask]

2. NumPy向量化函数

利用np.vectorize包装判断逻辑，底层基于NumPy优化，适合超大数据量场景：

import numpy as np

def keyword_in_description(desc, kw):
    return kw in desc

# 生成向量化判断函数
vec_judge = np.vectorize(keyword_in_description)
# 生成掩码并筛选
mask = vec_judge(df['Description'], df['keyword'])
filtered_df = df[mask]

3. Pandas apply方法（适合小数据量）

代码最易读，但逐行处理的特性导致大数据量下效率较低：

filtered_df = df[df.apply(lambda row: row['keyword'] in row['Description'], axis=1)]

注意事项 #

如果需要精确匹配完整单词（比如避免'play'匹配到'plays'），可以改用正则表达式的单词边界匹配，调整判断逻辑为：

import re

mask = [bool(re.search(r'\b' + re.escape(kw) + r'\b', desc)) for desc, kw in zip(df['Description'], df['keyword'])]

内容的提问来源于stack exchange，提问作者Christos Grigoriadis