在Pandas DataFrame中搜索组合关键词实现分类的问题

你的代码无法正确识别多词组合关键词，大概率是因为没考虑到短语匹配的格式统一性或精确性问题，以下两种方案可解决：

方案一：优化子串匹配逻辑 #

统一处理描述和关键词的格式，消除大小写、空格差异带来的匹配失效：

import pandas as pd

# 输入数据
data = {
    "job_description": [
        "Managing engineer is responsible for",
        "This job entails assisting to",
        "Engineer is required the execute",
        "Pilot should be able to control",
        "Customer specialist advices",
        "Different cases brought by human resources department"
    ]
}
df = pd.DataFrame(data)

cat_dict = {
    "manager": ["manager", "president", "management", "managing"],
    "assistant": ["assistant", "assisting", "customer specialist"],
    "engineer": ["engineer", "engineering", "scientist", "architect"],
    "HR": ["human resources"]
}

def classify(desc):
    # 统一转小写、去除首尾空格、合并多个空格为单个
    desc_clean = ' '.join(desc.lower().strip().split())
    for cat, lst in cat_dict.items():
        # 遍历关键词，转小写后匹配处理后的描述
        if any(keyword.lower() in desc_clean for keyword in lst):
            return cat
    # 无匹配项返回"other"
    return "other"

df['classification'] = df["job_description"].apply(classify)

方案二：正则表达式实现精确短语匹配 #

如果需要避免部分词误匹配（比如防止"human resources"被"human resources123"错误识别），可以用正则匹配完整短语：

import pandas as pd
import re

df = pd.DataFrame(data)

cat_dict = {
    "manager": ["manager", "president", "management", "managing"],
    "assistant": ["assistant", "assisting", "customer specialist"],
    "engineer": ["engineer", "engineering", "scientist", "architect"],
    "HR": ["human resources"]
}

def classify(desc):
    desc_lower = desc.lower()
    for cat, lst in cat_dict.items():
        for keyword in lst:
            # 构建正则规则：匹配完整短语，前后为单词边界或字符串首尾
            pattern = re.compile(r'\b' + re.escape(keyword.lower()) + r'\b')
            if pattern.search(desc_lower):
                return cat
    return "other"

df['classification'] = df["job_description"].apply(classify)

两种方案测试后，你的示例数据会得到预期分类结果，其中未匹配的"Pilot"类会被标记为"other"。

内容的提问来源于stack exchange，提问作者edyvedy13