Google Colab中spaCy NER自定义数据集训练自动终止求助

自定义数据集格式 #

已按spaCy要求调整自定义数据集格式，示例如下：

data[0]['text']示例：

RECEIVED REGISTER OF DEEDS KENT COUNTY, MI 2022 MAY 02 4:26 PM GU 51 202205030036938 Total Pages: 2 05/03/2022 11:00 AM Fees: $30.00 Lisa Posthumus Lyons, County Clerk/Register Kent County, MI SEAL QUIT CLAIM DEED 41-13-23-104-009 rc Debra Kathleen Hoek, as trustee of the Jeanette (Ma Janet) Hoek Living Trust u/a/d April 17, 2019, of 1058 Patton Avenue NW, Grand Rapids, Michigan 49504, QUIT CLAIMS to Janet Hoek,' individually, of 1058 Patton Avenue NW, Grand Rapids, Michigan 49504, the premises located in Kent County, Michigan, described as on the attached Exhibit A, subject to all easements and restrictions of record, for One Dollar ($1.00). This transfer is exempt from real estate transfer tax under MCLA 207.526(a), MSA 7.456(26) and MCLA 207.505(a), MSA 7.456(5). This conveyance does not create a division of any parcel of real property and no divisions have been made since March 31, 1997. This property may be located within the vicinity of 'farmland or a farm operation. Generally a..

data[0]['entities']示例：

[[70, 85, 'Recording Number'],
 [101, 111, 'Recording Date'],
 [199, 214, 'Doc Type'],
 [235, 311, 'Seller'],
 [405, 416, 'Buyer']]

复现步骤 #

1. 生成train.spacy文件，代码如下：

from spacy.util import filter_spans 

for training_example in tqdm(data): 
  text = training_example['text'] 
  labels = training_example['entities'] 
  doc = nlp.make_doc(text) 
  ents = [] 
  for start, end, label in labels: 
    span = doc.char_span(start, end, label=label, alignment_mode="contract") 
    if span is None: 
      print("Skipping entity") 
    else: 
      ents.append(span) 
      filtered_ents = filter_spans(ents) 
      doc.ents = filtered_ents 
      doc_bin.add(doc) 
  
  doc_bin.to_disk("train.spacy")

2. 生成config.cfg文件，执行命令：

!python -m spacy init fill-config base_config.cfg config.cfg

输出：

✔ Auto-filled config with all values
✔ Saved config
config.cfg
You can now add your data and train your pipeline:
python -m spacy train config.cfg --paths.train ./train.spacy --paths.dev ./dev.spacy

3. 执行训练命令：

!python -m spacy train config.cfg --output ./ --paths.train ./train.spacy --paths.dev ./train.spacy

输出显示训练初始化完成后自动终止（出现^C）：

ℹ Saving to output directory: .
ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  ------------  --------  ------  ------  ------  ------
^C

环境信息 #

• spaCy版本：3.7.3
• 平台：Linux-6.1.58+-x86_64-with-glibc2.35
• Python版本：3.10.12
• 预训练管道：en_core_web_lg (3.7.1), en_core_web_sm (3.7.1)

问题排查与解决建议 #

1. 修复数据集生成代码逻辑错误 #

原代码将filter_spans、doc.ents赋值和doc_bin.add(doc)放在实体循环内部，会导致每个实体都生成重复文档写入train.spacy，造成数据冗余甚至内存溢出。修正后代码：

from spacy.util import filter_spans 
from spacy.tokens import DocBin

doc_bin = DocBin()
nlp = spacy.blank("en")  # 或加载预训练模型

for training_example in tqdm(data): 
  text = training_example['text'] 
  labels = training_example['entities'] 
  doc = nlp.make_doc(text) 
  ents = [] 
  for start, end, label in labels: 
    span = doc.char_span(start, end, label=label, alignment_mode="contract") 
    if span is None: 
      print(f"Skipping entity at {start}-{end}: {text[start-10:end+10]}") 
    else: 
      ents.append(span) 
  # 所有实体处理完成后统一过滤并添加文档
  filtered_ents = filter_spans(ents) 
  doc.ents = filtered_ents 
  doc_bin.add(doc) 

doc_bin.to_disk("train.spacy")

2. 拆分训练集与验证集 #

训练命令中训练集和验证集混用会影响评估有效性，且可能触发异常。建议拆分独立验证集，或执行标准训练命令：

!python -m spacy train config.cfg --output ./ --paths.train ./train.spacy --paths.dev ./dev.spacy

3. 检查系统资源限制 #

若运行在共享资源环境（如Colab、云服务器），可能因CPU/内存配额不足导致进程被强制终止：

• 修改config.cfg中的batch_size参数，从默认64改为16/32减小内存占用
• 支持GPU的环境添加--gpu-id 0参数加速训练
• 查看系统日志（如dmesg）确认是否为内存不足（OOM）导致终止

4. 验证数据集完整性 #

生成train.spacy后，可执行以下代码验证数据是否正常：

import spacy
from spacy.tokens import DocBin

doc_bin = DocBin().from_disk("train.spacy")
nlp = spacy.blank("en")
docs = list(doc_bin.get_docs(nlp.vocab))
print(f"总文档数: {len(docs)}")
for doc in docs[:3]:
  print(f"文本长度: {len(doc.text)}, 实体: {[(ent.text, ent.label_) for ent in doc.ents]}")

内容的提问来源于stack exchange，提问作者Daremitsu