Presidio+Langchain Experimental无法识别波兰语姓名的问题求助

使用presidio/langchain_experimental对波兰语文本匿名化时，工具无法识别波兰语姓名（如"Jan Kowalski"），邮箱地址也未被处理。已通过python -m spacy download pl_core_news_lg安装对应模型，单独用spaCy加载该模型能正确识别实体；仅默认初始化PresidioReversibleAnonymizer()时功能正常，但自定义配置波兰语模型后失效。

代码示例 #

from presidio_anonymizer import PresidioAnonymizer
from presidio_reversible_anonymizer import PresidioReversibleAnonymizer

config = {
    "nlp_engine_name": "spacy",
    "models": [{"lang_code": "pl", "model_name": "pl_core_news_lg"}],
}

anonymizer = PresidioAnonymizer(analyzed_fields=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
                                languages_config=config)

anonymizer_tool = PresidioReversibleAnonymizer(analyzed_fields=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
                                               languages_config=config)

text = "Jan Kowalski mieszka w Warszawie i ma e-mail jan.kowalski@example.com."

anonymized_result = anonymizer_tool.anonymize(text)
anon_result = anonymizer.anonymize(text)
deanonymized_result = anonymizer_tool.deanonymize(anonymized_result)

print("Anonymized text:", anonymized_result)
print("Deanonymized text:", deanonymized_result)
print("Map:", anonymizer_tool.deanonymizer_mapping)
print("Anonymized text:", anon_result)

原因分析 #

核心问题是调用anonymize方法时未指定目标语言：Presidio默认使用英语（en）作为处理语言，即使配置了波兰语模型，工具仍会尝试用英语模型分析波兰语文本，自然无法识别波兰语姓名实体。

默认初始化PresidioReversibleAnonymizer()时，工具会自动加载多语言规则和内置模型，所以能识别部分实体，但这并不是基于你指定的spaCy波兰语模型的识别逻辑。

解决方案 #

在调用anonymize方法时显式添加language="pl"参数，确保Presidio使用你配置的波兰语spaCy模型进行实体分析：

修改后的代码 #

from presidio_anonymizer import PresidioAnonymizer
from presidio_reversible_anonymizer import PresidioReversibleAnonymizer

config = {
    "nlp_engine_name": "spacy",
    "models": [{"lang_code": "pl", "model_name": "pl_core_news_lg"}],
}

anonymizer = PresidioAnonymizer(analyzed_fields=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
                                languages_config=config)

anonymizer_tool = PresidioReversibleAnonymizer(analyzed_fields=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
                                               languages_config=config)

text = "Jan Kowalski mieszka w Warszawie i ma e-mail jan.kowalski@example.com."

# 关键：调用时指定language参数为波兰语
anonymized_result = anonymizer_tool.anonymize(text, language="pl")
anon_result = anonymizer.anonymize(text, language="pl")
deanonymized_result = anonymizer_tool.deanonymize(anonymized_result)

print("Anonymized text:", anonymized_result)
print("Deanonymized text:", deanonymized_result)
print("Map:", anonymizer_tool.deanonymizer_mapping)
print("Anonymized text:", anon_result)

额外验证方案（若问题仍存在） #

可以显式创建NlpEngine实例并传入匿名器，确保模型加载正确：

from presidio_analyzer import NlpEngineProvider

# 创建NlpEngine实例
provider = NlpEngineProvider(model_config=config)
nlp_engine = provider.create_engine()

# 初始化匿名器时传入nlp_engine
anonymizer = PresidioAnonymizer(
    analyzed_fields=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
    nlp_engine=nlp_engine
)

anonymizer_tool = PresidioReversibleAnonymizer(
    analyzed_fields=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
    nlp_engine=nlp_engine
)

这样可以强制工具使用你指定的波兰语spaCy模型，避免语言参数传递失误的问题。

内容的提问来源于stack exchange，提问作者Maltion