基于关键词层级对Pandas DataFrame职位描述分类的高效实现方法

原代码存在的问题 #

• 引用了不存在的列 cleaned，应替换为实际的job_description列
• 返回虚拟变量列，不符合需求的单一classification分类列格式
• 正则匹配未做大小写兼容，导致类似"Managing"的关键词无法匹配"managing"
• 逐行循环的实现方式处理大数据集时效率极低

高效分类实现方案 #

核心思路 #

严格遵循manager > assistant > engineer的优先级顺序，使用向量化操作快速匹配关键词：优先检查高优先级类别，匹配成功后直接跳过后续低优先级检查，确保分类结果符合层级规则。

推荐代码实现 #

import pandas as pd
import re
import numpy as np

# 定义带优先级的分类-关键词映射（顺序决定优先级）
category_keywords = [
    ("manager", ["manager", "president", "management", "managing"]),
    ("assistant", ["assistant", "assisting"]),
    ("engineer", ["engineer", "engineering", "scientist", "architect"])
]

# 示例DataFrame
data = {
    "job_description": [
        "Managing engineer is responsible for",
        "This job entails assisting to engineers",
        "Engineer is required to execute",
        "Pilot should be able to control"
    ]
}
df = pd.DataFrame(data)

# 过滤缺失职位描述的行
df2 = df[~df['job_description'].isna()].copy()

# 构建匹配条件与对应分类
conditions = []
choices = []
for category, keywords in category_keywords:
    # 构建安全正则表达式（转义特殊字符），匹配完整单词且不区分大小写
    pattern = r'\b(' + '|'.join(re.escape(k) for k in keywords) + r')\b'
    conditions.append(df2['job_description'].str.contains(pattern, case=False, regex=True))
    choices.append(category)

# 应用优先级匹配，未匹配项设为None
df2['classification'] = np.select(conditions, choices, default=None)

# 输出结果
print(df2)

关键优化点 #

• 向量化操作：用pandas.str.contains和np.select替代逐行循环，处理十万级以上数据时效率提升数十倍
• 正则安全性：通过re.escape()处理关键词中的特殊字符，避免正则语法错误
• 大小写兼容：case=False确保匹配不受大小写格式影响
• 优先级保障：np.select会优先匹配第一个满足的条件，完全符合层级规则

执行结果 #

job_description classification
0  Managing engineer is responsible for         manager
1     This job entails assisting to engineers      assistant
2        Engineer is required to execute        engineer
3         Pilot should be able to control           None

内容的提问来源于stack exchange，提问作者edyvedy13