spaCy自定义NER模型训练后无法识别新数据集实体问题

问题背景 #

我正尝试针对自身业务场景构建若干自定义NER模型。

训练数据样本 #

[[' webex enable or disable auto join from connected video device advised user to raise a iiq request to get webex access.  hence closing this ticket getting error while joing webex since i got a replacement lap top i have not been able to access webex 954 995 0444 9 30am 6pm thurs mon getting error while joing webex.',
  {'entities': [[1, 62, 'ISSUE'],
    [63, 118, 'RESOLVE_ACTION'],
    [147, 178, 'ISSUE'],
    [193, 212, 'USER_ACTIVITY']]}],
 [' virtual desktop infrastructure vdi performance or latency issues or slow response education vdi user was getting vdi slowness issue while working on it.  i have guide to user kindly logoff your vdi machine and login back after 10 minutes.  user was getting vdi slowness issue while working on it.  vdi slowness.',
  {'entities': [[1, 47, 'ISSUE'],
    [51, 65, 'ISSUE'],
    [69, 96, 'ISSUE'],
    [183, 238, 'RESOLVE_ACTION']]}]]

现有实现代码 #

ner = nlp.get_pipe("ner")
for _, annotations in train_data:
    for ent in annotations.get('entities'):
        ner.add_label(ent[2])
disable_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
import random 
from spacy.util import minibatch, compounding
from pathlib import Path

with nlp.disable_pipes(*disable_pipes):
    optimizer =nlp.resume_training()
    
    for iteration in range(100):
        
        random.shuffle(train_data)
        losses = {}
        
        batches = minibatch(train_data,size=compounding(1, 16, 1.001))
        for batch in batches:
            text,annotation = zip(*batch)
            print (text,annotation)
            nlp.update(text,annotation,drop=0.5,losses=losses,sgd=optimizer)
            print("Losses",losses)

output_dir = '/<my model path>/models'
if output_dir is not None:
    output_dir = Path('/my model path>/models')
    if not output_dir.exists():
        output_dir.mkdir()
    nlp.meta['name'] = "new_model"  # rename model
    nlp.to_disk(output_dir)
    print("Saved model to", output_dir)

#Testing model on training data set itself

for text, _ in train_data:
    doc = nlp2(text)
    print ('Entities',[(ent.text,ent.label_) for ent in doc.ents])
    displacy.render(nlp2(doc.text), style="ent", jupyter=True)

故障现象 #

模型在训练数据集上可正常返回对应实体标签，但在其余样本/文本上无法识别返回任何实体结果。

核心问题点 #

• 训练数据量严重不足：当前仅用2条标注样本训练，模型只能死记硬背训练集内容，完全无法学习到通用的实体特征，自定义NER至少需要50条以上覆盖不同表述、不同上下文的标注样本，才能具备基础泛化能力。
• 标注索引存在偏移：spaCy的实体标注要求索引从0开始、遵循左闭右开规则，当前所有标注的起始索引为1，实体边界整体错位，模型学到的实体判定规则本身就是错误的。
• 训练参数不合理：
  1. compounding(1, 16, 1.001) 增长速率过低，训练全程几乎都用batch size=1更新参数，梯度波动极大，模型收敛效果差
  2. 小数据集下dropout设置为0.5过高，会随机丢弃一半的特征，进一步干扰模型学习有效规律，小数据集建议设置为0.2~0.3
  3. 仅2条样本的情况下迭代100轮会导致严重过拟合，模型完全拟合训练集噪声，泛化能力归零
• 代码逻辑错误：训练时使用nlp对象，测试时调用未定义/加载旧模型的nlp2对象；模型保存路径存在拼写错误（多了一个多余的>符号），实际保存的模型可能和测试加载的模型不一致。

修复后参考代码 #

import random
from spacy.util import minibatch, compounding
from pathlib import Path
import spacy
from spacy import displacy

# 优先加载带词向量的预训练模型（如en_core_web_md），可大幅提升小样本下的泛化能力
nlp = spacy.load("en_core_web_sm")
ner = nlp.get_pipe("ner")

# 第一步：修正所有标注的索引为0起始，做数据校验
# 校验逻辑：对每条标注，用text[start:end]切片，确认切出的内容和预期实体完全一致
train_data = [
    # 替换为修正索引后的训练数据
]
for _, annotations in train_data:
    for ent in annotations.get("entities"):
        ner.add_label(ent[2])

# 冻结NER以外的所有管道，减少无关参数干扰
disable_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
with nlp.disable_pipes(*disable_pipes):
    optimizer = nlp.resume_training()
    # 小数据集迭代轮次控制在20~30即可，避免过拟合
    for iteration in range(30):
        random.shuffle(train_data)
        losses = {}
        # 调整batch增长逻辑，从4起步逐步涨到32，保证梯度稳定
        batches = minibatch(train_data, size=compounding(4, 32, 1.001))
        for batch in batches:
            text, annotation = zip(*batch)
            # 调低dropout比例
            nlp.update(text, annotation, drop=0.3, losses=losses, sgd=optimizer)
        print(f"Iteration {iteration}, Losses: {losses}")

# 修正路径拼写错误
output_dir = Path("./custom_ner_models")
output_dir.mkdir(parents=True, exist_ok=True)
nlp.meta["name"] = "custom_ner"
nlp.to_disk(output_dir)
print(f"Saved model to {output_dir}")

# 测试时正确加载训练完成的模型
nlp_test = spacy.load(output_dir)
for text, _ in train_data:
    doc = nlp_test(text)
    print("Entities", [(ent.text, ent.label_) for ent in doc.ents])
    displacy.render(doc, style="ent", jupyter=True)

优化建议 #

• 标注完成后必须做全量数据校验，逐一切片核对实体文本和边界，避免索引错位、实体漏标/错标问题。
• 扩充数据集时尽量覆盖实体的不同表述、不同上下文场景，比如ISSUE类实体不要仅标注Webex报错、VDI卡顿，还要覆盖其他软件、其他故障类型的文本。
• 训练时拆分训练集和验证集，每轮迭代后在验证集上测试精确率、召回率，一旦出现训练集效果上升、验证集效果下降的过拟合情况，及时减少迭代轮次、补充训练数据。

内容的提问来源于stack exchange，提问作者arun