训练Spark NLP自定义NER模型：spaCy Doc转CoNLL2003方案问询

我计划训练一个Spark NLP自定义NER模型，该模型要求采用CoNLL 2003格式进行训练，但我有自己的训练数据（由一系列spaCy Doc对象组成），不清楚如何将其转换为CoNLL 2003格式。目前我尝试了三种方法，每种都存在明显缺陷：

• spacy_conll项目：文档中的示例使用空白spaCy模型，不清楚如何融入自己的标注数据；另外conll_formatter组件似乎是添加在管道末尾，不确定它是否直接实现了Doc到CoNLL的转换。
• Prodigy论坛（spaCy同团队产品）中的方案：生成的CoNLL 2003格式不完整，缺少POS标签（可通过Token.pos_获取）以及句法块标签（spaCy中无对应属性），而这两个字段是CoNLL 2003官方文档规定的必填字段。
• textacy库的转换实现：从0.11.0版本开始被弃用，当前最新版本为0.12.0，不确定该方案是否仍可使用。

我当前的代码如下：

import spacy
from spacy.training import offsets_to_biluo_tags
from spacy.tokens import Span

print("SPACY HELPER MODEL")
base_model = "en_core_web_sm"
nlp = spacy.load(base_model)
to_disable= ['parser', 'lemmatizer', 'ner']
_ = [nlp.remove_pipe(item) for item in to_disable]
print("Base model used: ", base_model)
print("Removed components: ", to_disable)
print("Enabled components: ", nlp.pipe_names)

# Assume text is already available as sentences...
# so no need for spaCy `sentencizer` or similar
print("\nDEMO SPACY DOC LIST BUILDING...", end="")
doc1 = nlp("iPhone X is coming.")
doc1.ents = [Span(doc1, 0, 2, label="GADGET")]
doc2 = nlp("Space X is nice.")
doc2.ents = [Span(doc1, 0, 2, label="BRAND")]
docs = [doc1, doc2]
print("DONE!")

print("\nCoNLL 2003 CONVERSION:\n")
results = []
for doc in docs:
    # Preliminary: whole sentence
    whole_sentence = doc.text
    # 1st item (CoNLL 2003): word
    words = [token.text for token in doc]
    # 2nd item (CoNLL 2003): POS
    pos = [token.tag_ for token in doc]
    # 3rd item (CoNLL 2003): syntactic chunk tag
    sct = ["[UNKNOWN]" for token in doc]
    # 4th item (CoNLL 2003): named entities
    spacy_entities = [
        (ent.start_char, ent.end_char, ent.label_)
        for ent in doc.ents
    ]
    biluo_entities = offsets_to_biluo_tags(doc, spacy_entities)
    results.append((whole_sentence, words, pos, sct, biluo_entities))

for result in results:
    print(
        "\nDOC TEXT (NOT included in CoNLL 2003, just for demo): ",
        result[0], "\n"
    )
    print("-DOCSTART- -X- -X- O")
    for w,x,y,z in zip(result[1], result[2], result[3], result[4]):
        print(w,x,y,z)

# Pending: write to a file, but that's easy, and out of topic.

代码输出如下：

DOC TEXT (NOT included in CoNLL 2003, just for demo):  iPhone X is coming.

-DOCSTART- -X- -X- O
iPhone NNP [UNKNOWN] B-GADGET
X NNP [UNKNOWN] L-GADGET
is VBZ [UNKNOWN] O
coming VBG [UNKNOWN] O
. . [UNKNOWN] O

DOC TEXT (NOT included in CoNLL 2003, just for demo):  Space X is nice.

-DOCSTART- -X- -X- O
Space NNP [UNKNOWN] B-BRAND
X NNP [UNKNOWN] L-BRAND
is VBZ [UNKNOWN] O
nice JJ [UNKNOWN] O
. . [UNKNOWN] O

请问是否有人做过类似的转换？求可行的解决方案。

内容的提问来源于stack exchange，提问作者David Espinosa