如何使用Python修改JSONL文件中spans的label值？

问题描述 #

你有一个Prodigy标注生成的JSONL文件，示例内容如下：

{
  "text": "More specifically, Haile 16A was suggested as most likely deriving from the early half of the late early Irvingtonian (1.6 to 1.3 Ma) by the use of mammalian biochronology (Morgan and Hulbert, 1995).",
  "spans": [
    {
      "start": 148,
      "end": 171,
      "token_start": 30,
      "token_end": 31,
      "label": "ACT_IMPLIED"
    }
  ]
}

希望将spans数组中label字段为"ACT_IMPLIED"的值修改为"ACTIVITY"，但使用Pandas编写的代码未生效：

import pandas as pd

path="/content/sample.jsonl"
data = pd.read_json(path, lines=True)

for idx, row in data.iterrows():
    for span in row.spans:
        if span['label'] == 'ACT_IMPLIED':
          span["label"].replace('ACT_IMPLIED', "ACTIVITY")

问题原因 #

你的代码中使用了str.replace()方法，但该方法返回修改后的新字符串，不会直接修改原变量的值。因此span["label"].replace(...)并没有改变span["label"]的实际内容。

修正方案 #

方案1：直接赋值（修复原代码） #

既然已经判断span['label'] == 'ACT_IMPLIED'，直接将span["label"]赋值为"ACTIVITY"即可：

import pandas as pd

path="/content/sample.jsonl"
data = pd.read_json(path, lines=True)

for idx, row in data.iterrows():
    for span in row.spans:
        if span['label'] == 'ACT_IMPLIED':
            span["label"] = "ACTIVITY"  # 直接替换赋值

# 保存修改后的结果到新文件
data.to_json("/content/modified_sample.jsonl", orient='records', lines=True)

方案2：使用apply提高效率（推荐） #

对于大型数据集，iterrows效率较低，推荐使用apply方法批量处理每行的spans：

import pandas as pd

path="/content/sample.jsonl"
data = pd.read_json(path, lines=True)

def update_spans_labels(spans):
    for span in spans:
        if span['label'] == 'ACT_IMPLIED':
            span['label'] = 'ACTIVITY'
    return spans

data['spans'] = data['spans'].apply(update_spans_labels)

# 保存结果
data.to_json("/content/modified_sample.jsonl", orient='records', lines=True)

验证结果 #

运行代码后，生成的modified_sample.jsonl文件中对应的label值会被修改为"ACTIVITY"：

{
  "text": "More specifically, Haile 16A was suggested as most likely deriving from the early half of the late early Irvingtonian (1.6 to 1.3 Ma) by the use of mammalian biochronology (Morgan and Hulbert, 1995).",
  "spans": [
    {
      "start": 148,
      "end": 171,
      "token_start": 30,
      "token_end": 31,
      "label": "ACTIVITY"
    }
  ]
}

内容的提问来源于stack exchange，提问作者mar_k