Python处理越南姓名数据清洗的技术咨询

问题背景 #

我正在处理一个包含20万条越南姓名的混乱数据库，核心问题是姓名字段混杂了大量中间名和首字母，且越南姓名的语法规则导致通用解析工具失效。示例问题姓名：

• Anh Quoc Hoang
• Anh Tuan Dao
• Anh Tuyet Thi

目前使用from nameparser import HumanName处理，但该工具对越南姓名解析效果极差，加上对越南姓名文化了解不足，大量边缘案例只能直接跳过。

当前处理脚本 #

import glob
import pandas as pd
import numpy as np
import os
from nameparser import HumanName

extended_last_names = [
    "Nguyen", "Tran", "Le", "Pham", "Hoang", "Huynh", "Phan", "Vu", "Vo",
    "Dang", "Bui", "Do", "Ho", "Ngo", "Duong", "Ly", "Luu", "Quach", "Trinh",
    "Phung", "Dinh", "Doan", "Dai", "Giang", "Ha", "Han", "Kieu", "Lai",
    "Lam", "Luong", "Mai", "Nghiem", "Phi", "Quang", "Quoc", "Ta", "Thach",
    "Thai", "Thao", "Thieu", "Tho", "Tong", "Trang", "Trieu", "Truong",
    "Tuan", "Van", "Vinh", "Vuong", "Xuan", "Thanh", "Tung", "Quyen", "Chau",
    "Kha", "Khanh", "Minh", "Nam", "Quan", "Hieu", "Hai", "Hien", "Hung",
    "Huong", "Khang", "Khoi", "Linh", "Nhat", "Quynh", "Son", "Thuy", "Tien",
    "Anh", "Bach", "Bang", "Binh", "Chien", "Cong", "Cuong", "Duy", "Gia",
    "Hao", "Kiet", "Loc", "Long", "Nhan", "Phuc", "Sang", "Tam", "Thang",
    "Thien", "Toan", "Trung", "Tuyet", "Vien", "Yen", "Wang", "Li", "Zhang",
    "Liu", "Chen", "Yang", "Huang", "Zhao", "Wu", "Zhou", "Xu", "Sun", "Ma",
    "Zhu", "Hu", "Guo", "He", "Gao", "Lin", "Luo"
]

# Function to parse a name
def parse_name(first_name, last_name):
    try:
        # If the last name is in the extended list, ignore the first name
        if last_name in extended_last_names:
            return np.nan, np.nan, last_name
        
        # Parse the first name using HumanName
        parsed = HumanName(first_name)
        # Return the parsed names
        return parsed.first, parsed.middle, parsed.last
    except Exception as e:
        print(f"Error parsing name '{first_name} {last_name}': {e}")
        return np.nan, np.nan, np.nan

# Any file ending in csv
files = glob.glob('*.csv')
for file in files:
    # Skip files with '_clean' in the name
    if '_clean' in file:
        continue

    df = pd.read_csv(file, encoding='ISO-8859-1', low_memory=False)

    # Apply the function and create new columns for first, middle, and last names
    df[['CnBio_First_Name', 'CnBio_Middle_Name', 'CnBio_Last_Name']] = df.apply(
        lambda x: pd.Series(parse_name(x['CnBio_First_Name'], x['CnBio_Last_Name'])), axis=1)
    df[['CnSpSpBio_First_Name', 'CnSpSpBio_Middle_Name', 'CnSpSpBio_Last_Name']] = df.apply(
        lambda x: pd.Series(parse_name(x['CnSpSpBio_First_Name'], x['CnSpSpBio_Last_Name'])), axis=1)

    base, ext = os.path.splitext(file)

    # Insert '_clean' before the extension
    new_file = base + '_clean' + ext
    # Save the DataFrame to the new file name in the current working directory
    df.to_csv(f'{new_file}', index=False, encoding='ISO-8859-1')

优化越南姓名解析的核心思路 #

1. 明确越南姓名结构规则 #

越南姓名严格遵循**「姓 + 中间名 + 名」**的顺序，核心特点：

• 姓氏多为单字（如Nguyen、Tran），复姓极其罕见
• 中间名可包含1到多个词，部分是固定尊称（如Van、Thi）
• 名是姓名的最后一个词，是日常称呼的核心

2. 重构解析逻辑：从后往前匹配姓氏 #

放弃通用解析工具，改用针对性策略：

• 将完整姓名按空格拆分为词列表
• 从最后一个词开始往前遍历，匹配越南姓氏列表，找到第一个匹配项作为姓氏
• 剩余词中，最后一个是名，中间所有词为中间名

3. 优化姓氏列表，剔除无关项 #

移除混入的中文姓氏，保留越南本土高频姓氏：

vietnamese_last_names = [
    "Nguyen", "Tran", "Le", "Pham", "Hoang", "Huynh", "Phan", "Vu", "Vo",
    "Dang", "Bui", "Do", "Ho", "Ngo", "Duong", "Ly", "Luu", "Quach", "Trinh",
    "Phung", "Dinh", "Doan", "Dai", "Giang", "Ha", "Han", "Kieu", "Lai",
    "Lam", "Luong", "Mai", "Nghiem", "Phi", "Quang", "Ta", "Thach", "Thai",
    "Thao", "Thieu", "Tho", "Tong", "Trang", "Trieu", "Truong", "Van", "Vinh",
    "Vuong", "Xuan", "Thanh", "Tung", "Quyen", "Chau", "Kha", "Khanh", "Minh",
    "Nam", "Quan", "Hieu", "Hai", "Hien", "Hung", "Huong", "Khang", "Khoi",
    "Linh", "Nhat", "Quynh", "Son", "Thuy", "Tien", "Anh", "Bach", "Bang",
    "Binh", "Chien", "Cong", "Cuong", "Duy", "Gia", "Hao", "Kiet", "Loc",
    "Long", "Nhan", "Phuc", "Sang", "Tam", "Thang", "Thien", "Toan", "Trung",
    "Tuyet", "Vien", "Yen"
]

4. 重写解析函数，适配越南姓名 #

def parse_vietnamese_name(full_name):
    if pd.isna(full_name) or full_name.strip() == "":
        return np.nan, np.nan, np.nan
    
    name_parts = full_name.strip().split()
    if len(name_parts) == 0:
        return np.nan, np.nan, np.nan
    
    # 从后往前匹配姓氏
    last_name = None
    name_idx = len(name_parts) - 1
    while name_idx >= 0:
        if name_parts[name_idx] in vietnamese_last_names:
            last_name = name_parts[name_idx]
            break
        name_idx -= 1
    
    if not last_name:
        # 未匹配到姓氏时，默认第一个词为姓（越南姓名姓在前概率极高）
        last_name = name_parts[0]
        given_parts = name_parts[1:]
    else:
        given_parts = name_parts[:name_idx]
    
    # 拆分名字和中间名
    if len(given_parts) == 0:
        first_name = last_name
        middle_name = np.nan
    elif len(given_parts) == 1:
        first_name = given_parts[0]
        middle_name = np.nan
    else:
        first_name = given_parts[-1]
        middle_name = " ".join(given_parts[:-1])
    
    return first_name, middle_name, last_name

5. 调整主逻辑，修复字段拆分混乱问题 #

原字段的First_Name和Last_Name可能本身拆分错误，建议拼接后再解析：

def process_name_columns(row, prefix):
    # 拼接原字段得到完整姓名
    full_name = f"{row[f'{prefix}_First_Name']} {row[f'{prefix}_Last_Name']}".strip()
    return pd.Series(parse_vietnamese_name(full_name))

# 替换原apply逻辑
df[['CnBio_First_Name', 'CnBio_Middle_Name', 'CnBio_Last_Name']] = df.apply(
    lambda x: process_name_columns(x, 'CnBio'), axis=1)
df[['CnSpSpBio_First_Name', 'CnSpSpBio_Middle_Name', 'CnSpSpBio_Last_Name']] = df.apply(
    lambda x: process_name_columns(x, 'CnSpSpBio'), axis=1)

6. 边缘案例处理 #

• 单字姓名：直接作为姓+名，中间名留空
• 无法匹配姓氏的姓名：默认第一个词为姓，剩余部分归为名字
• 含尊称的姓名：尊称自动归入中间名，无需特殊处理

内容的提问来源于stack exchange，提问作者feelsgood