迭代修改Span列表：NER任务生成Dummy Data遇重复实体异常问题

我正在为NER任务生成虚拟数据，需要把文本中的person_name实体替换为虚拟名称，但同一实体多次出现时，Span列表迭代修改会产生异常结果。

输入示例 #

{ 
  "text": "Mohan dob is 25th dec 1980. Mohan loves to play cricket.", 
  "spans": [
    {"start":0, "end":5,"label":"person_name","ngram":"Mohan"}, 
    {"start":28, "end":33,"label":"person_name","ngram":"Mohan"}, 
    {"start":13, "end":26,"label":"date","ngram":"25th dec 1980"}
  ] 
}

样本中person_name实体出现两次，sample_names=['Jon', 'Sam']，需要将两个person_name实体替换为列表中的名称，同时更新对应的Span信息。

预期输出 #

[
  {
    "text": "Jon dob is 25th dec 1980. Jon loves to play cricket.",
    "spans": [
      {"start":0, "end":3,"label":"person_name","ngram":"Jon"},
      {"start":11, "end":24,"label":"date","ngram":"25th dec 1980"},
      {"start":26, "end":31,"label":"person_name","ngram":"Jon"}
    ]
  },
  {
    "text": "Sam dob is 25th dec 1980. Sam loves to play cricket.",
    "spans": [
      {"start":0, "end":3,"label":"person_name","ngram":"Sam"},
      {"start":11, "end":24,"label":"date","ngram":"25th dec 1980"},
      {"start":26, "end":31,"label":"person_name","ngram":"Sam"}
    ]
  }
]

当前使用的代码 #

def generate(data, target_entity, names):
    text = data['text']
    spans = data['spans']
    new_sents=[]

    if spans:
        spans = [(d['start'], d['end'], d['label']) for d in spans]
        spans.sort()
        labellist=[s[2] for s in spans]
        # get before_spans and after_spans around target entity
        
        for n in names:
            gap = 0
            for i, tup in enumerate(spans):
                lab = tup[2]
                if lab == target_entity: 
                    
                    new_spans={"before": spans[:i], "after": spans[i+1:]}
                    print("the spans before and after  :\n",new_spans)
                    start=tup[0] #check this
                    end=tup[1]
                    ngram = text[start:end]
                    new_s = text[:start] + n + text[end:]
                    gap = len(n) - len(ngram)
                    
                    before = new_spans["before"]
                    after = [(tup[0]+gap, tup[1]+gap, tup[2]) for tup in new_spans["after"]]
                    s_sp = before + [(start, start + len(n), target_entity)] + after
                    text=new_s
                    en={"text": new_s,"spans": [{"start": tup[0], "end": tup[1], "label": tup[2], "ngram": new_s[tup[0]:tup[1]]} for tup in s_sp]}
                    spans = s_sp
                    
            new_sents.append(en)

问题分析与修正方案 #

原代码的核心问题是：

1. 处理每个虚拟名称时，复用了前一次修改后的文本和Span，导致位置计算混乱
2. 迭代修改Span列表时，没有正确累积文本长度变化带来的偏移量

修正后的代码思路：

• 每次处理新虚拟名称时，都基于原始文本和原始Span开始，避免交叉污染
• 用累积偏移量跟踪文本长度变化，统一更新所有Span的位置
• 分开处理目标实体和非目标实体，最后按位置排序保证Span顺序正确

修正后的代码 #

def generate(data, target_entity, names):
    original_text = data['text']
    # 原始Span按start位置排序，确保处理顺序正确
    original_spans = sorted(
        [(d['start'], d['end'], d['label'], d['ngram']) for d in data['spans']],
        key=lambda x: x[0]
    )
    new_sents = []

    for name in names:
        current_text = original_text
        offset = 0
        updated_spans = []
        
        # 分离目标实体和非目标实体
        target_positions = []
        non_target_spans = []
        for span in original_spans:
            start, end, label, ngram = span
            if label == target_entity:
                target_positions.append((start, end, ngram))
            else:
                non_target_spans.append((start, end, label, ngram))
        
        # 按顺序替换目标实体，累积偏移量
        temp_text = current_text
        for start, end, ngram in target_positions:
            adjusted_start = start + offset
            adjusted_end = end + offset
            # 替换实体文本
            temp_text = temp_text[:adjusted_start] + name + temp_text[adjusted_end:]
            # 更新偏移量：新名称与原名称的长度差
            delta = len(name) - len(ngram)
            offset += delta
            # 添加替换后的Span
            updated_spans.append({
                "start": adjusted_start,
                "end": adjusted_start + len(name),
                "label": target_entity,
                "ngram": name
            })
        
        # 更新非目标实体的Span位置
        for start, end, label, ngram in non_target_spans:
            adjusted_start = start + offset
            adjusted_end = end + offset
            updated_spans.append({
                "start": adjusted_start,
                "end": adjusted_end,
                "label": label,
                "ngram": temp_text[adjusted_start:adjusted_end]
            })
        
        # 按start位置重新排序Span，保持顺序一致性
        updated_spans.sort(key=lambda x: x['start'])
        new_sents.append({
            "text": temp_text,
            "spans": updated_spans
        })
    
    return new_sents

# 测试用例
sample_data = { 
  "text": "Mohan dob is 25th dec 1980. Mohan loves to play cricket.", 
  "spans": [
    {"start":0, "end":5,"label":"person_name","ngram":"Mohan"}, 
    {"start":28, "end":33,"label":"person_name","ngram":"Mohan"}, 
    {"start":13, "end":26,"label":"date","ngram":"25th dec 1980"}
  ] 
}
sample_names = ['Jon', 'Sam']
result = generate(sample_data, 'person_name', sample_names)
import json
print(json.dumps(result, indent=2))

内容的提问来源于stack exchange，提问作者MAC