使用SpaCy提取GPE实体误识别非地理地点的解决方案咨询

问题背景 #

使用SpaCy提取文本中的地理地点（GPE实体）时，出现误识别情况，像"BA297"、"Legal"、"Nutrigain"、"Arham"、"Stephen"这类非地理实体被标记为GPE。当前使用的代码如下：

# Load spaCy English model
nlp = spacy.load("en_core_web_sm")

# Perform named entity recognition (NER) using spaCy
entities = []
for word in df['words']:
    doc = nlp(word)
    for ent in doc.ents:
        entities.append((ent.text, ent.label_))

# Extract GPE entities
gpe_entities = [entity[0] for entity in entities if entity[1] == 'GPE']

基于SpaCy的优化方案 #

1. 切换更大规模的预训练模型 #

en_core_web_sm是轻量模型，识别精度有限。换成en_core_web_md或en_core_web_lg，这类模型训练数据更丰富，实体识别准确率更高：

# 替换为中/大规模模型
nlp = spacy.load("en_core_web_lg")

2. 自定义规则修正实体识别 #

使用SpaCy的EntityRuler添加规则，排除误识别的词汇或纠正实体标签：

import spacy
from spacy.pipeline import EntityRuler

nlp = spacy.load("en_core_web_sm")
# 初始化EntityRuler，设置 overwrite_ents=True 覆盖原有标签
ruler = EntityRuler(nlp, overwrite_ents=True)

# 添加规则：将错误识别的词标记为正确类型或排除
patterns = [
    {"label": "PERSON", "pattern": "Stephen"},
    {"label": "PERSON", "pattern": "Arham"},
    {"label": "PRODUCT", "pattern": "Nutrigain"},
    {"label": "ORG", "pattern": "Legal"},
    {"label": "OTHER", "pattern": "BA297"}
]
ruler.add_patterns(patterns)
# 将ruler添加到pipeline
nlp.add_pipe(ruler, before="ner")

# 后续识别逻辑不变
entities = []
for word in df['words']:
    doc = nlp(word)
    for ent in doc.ents:
        entities.append((ent.text, ent.label_))

gpe_entities = [entity[0] for entity in entities if entity[1] == 'GPE']

替代库方案 #

1. Hugging Face Transformers（高精度，支持微调） #

使用预训练的NER模型，可针对自定义数据微调，大幅提升识别精度：

from transformers import pipeline

# 加载预训练NER模型
ner_pipeline = pipeline("ner", model="dbmdz/bert-large-cased-finetuned-conll03-english")

gpe_entities = []
for word in df['words']:
    results = ner_pipeline(word)
    # 筛选GPE（对应标签为I-LOC或B-LOC，不同模型标签可能有差异）
    for res in results:
        if res['entity'].startswith('LOC'):
            gpe_entities.append(res['word'])

2. NLTK（轻量，适合简单场景） #

结合NLTK的NER工具，可搭配自定义地理词典辅助识别：

import nltk
from nltk.tag import pos_tag
from nltk.chunk import ne_chunk

nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')

gpe_entities = []
for word in df['words']:
    tokens = nltk.word_tokenize(word)
    tagged = pos_tag(tokens)
    chunks = ne_chunk(tagged)
    for chunk in chunks:
        if hasattr(chunk, 'label') and chunk.label() == 'GPE':
            gpe_entities.append(' '.join(c[0] for c in chunk))

内容的提问来源于stack exchange，提问作者Ankit