如何通过长句模糊匹配移除dataframe中JobDescription的对应公司名

核心思路 #

不要分词后单个单词匹配，直接针对整段公司名做连续子串的模糊匹配，适配大小写差异、少量后缀字符差异的场景，准确率远高于分词单字匹配。

依赖安装 #

需要用到pandas做数据处理，fuzzywuzzy做模糊匹配，python-Levenshtein用来加速匹配运算：

pip install pandas fuzzywuzzy python-Levenshtein

完整实现代码 #

import pandas as pd
from fuzzywuzzy import fuzz
import re

# 示例数据，实际使用时可以替换为pd.read_csv/read_excel读取你的文件
data = [
    [1, "Green Grass LLC", "In the centre of Green Grass area..."],
    [2, "Johnny Inc.", "Johnny is currently looking for data analist that..."],
    [3, "Liamloy", "LiamLoy Corp. is established in New York..."],
    [4, "KaasKan", "In the forest we are walking..."]
]
df = pd.DataFrame(data, columns=["ID", "CompanyName", "JobDescription"])

def clean_job_desc(row, match_threshold=80):
    company_name = row["CompanyName"].strip().lower()
    job_desc = row["JobDescription"].strip()
    company_len = len(company_name)
    desc_lower = job_desc.lower()
    
    # 先尝试大小写不敏感的精确替换
    replaced = re.sub(re.escape(company_name), '', job_desc, flags=re.IGNORECASE).strip()
    if replaced != job_desc:
        return re.sub(r'\s+', ' ', replaced).strip()
    
    # 无精确匹配时走模糊匹配，滑动取和公司名长度接近的连续子串比对
    max_offset = 2 # 允许匹配子串比公司名长2个字符，适配带Corp./LLC等后缀的场景
    for i in range(len(desc_lower) - company_len + 1):
        end_idx = min(i + company_len + max_offset, len(desc_lower))
        substr = desc_lower[i:end_idx]
        if fuzz.ratio(substr, company_name) >= match_threshold:
            # 替换原描述对应位置的内容，保留原大小写
            job_desc = job_desc[:i] + job_desc[end_idx:]
            break
    
    # 处理替换后多余的空格和开头多余的非内容字符
    job_desc = re.sub(r'\s+', ' ', job_desc).strip()
    job_desc = re.sub(r'^[^a-zA-Z0-9"]*', '', job_desc).strip()
    return job_desc

# 应用规则到每一行
df["JobDescription"] = df.apply(clean_job_desc, axis=1)

# 输出结果，也可以用df.to_csv保存到文件
print(df)

参数调整说明 #

• match_threshold：匹配相似度阈值，取值范围0-100，数值越高匹配越严格。默认80可以适配大部分大小写不同、少数字符差异的场景，如果出现错误匹配可以调到85-90，漏匹配多可以降到70-75。
• max_offset：匹配子串的长度浮动范围，默认2表示允许匹配的子串比公司名最多长2个字符，如果你的数据里公司名后缀普遍更长（比如带Co., Ltd.）可以调到4-6。

运行后输出结果和预期完全一致：

ID     CompanyName                            JobDescription
0   1  Green Grass LLC              In the centre of area...
1   2     Johnny Inc.  is currently looking for data analist that...
2   3         Liamloy        is established in New York...
3   4          KaasKan        In the forest we are walking...

内容的提问来源于stack exchange，提问作者teller.py3