使用spacyr进行命名实体识别结果不一致的问题求助

我想用R的spacyr库（Python spaCy的R封装）对新闻文章做命名实体识别（NER），目标是自动识别合作伙伴来开展网络分析，但spacyr未按预期识别常见实体。

示例代码 #

library(quanteda)
library(spacyr)

text <- data.frame(doc_id = c(1:5),
                   sentence = c("Brightmark LLC, the global waste solutions provider, and Florida Keys National Marine Sanctuary (FKNMS), today announced a new plastic recycling partnership that will reduce landfill waste and amplify concerns about ocean plastics.",
                                "Brightmark is launching a nationwide site search for U.S. locations suitable for its next set of advanced recycling facilities, which will convert hundreds of thousands of tons of post-consumer plastics into new products, including fuels, wax, and other products.",
                                "Brightmark will be constructing the facility in partnership with the NSW government, as part of its commitment to drive economic growth and prosperity in regional NSW.",
                                "Macon-Bibb County, the Macon-Bibb County Industrial Authority, and Brightmark have mutually agreed to end discussions around building a plastic recycling plant in Macon",
                                "Global petrochemical company SK Global Chemical and waste solutions provider Brightmark have signed a memorandum of understanding to create a partnership that aims to take the lead in the circular economy of plastic by construction of a commercial scale plastics renewal plant in South Korea"))

corpus <- corpus(text, text_field = "sentence")

spacy_initialize(model = "en_core_web_sm")

parsed <- spacy_parse(corpus)

entity <- entity_extract(parsed)

预期与实际结果 #

我预期「Brightmark」在5个句子中都被识别为ORG类型，「NSW government」也被识别为ORG，但实际输出的entity结果如下：

doc_id sentence_id                                 entity entity_type
1      1           1 Florida_Keys_National_Marine_Sanctuary         ORG
2      1           1                                  FKNMS         ORG
3      2           1                                   U.S.         GPE
4      3           1                                    NSW         ORG
5      4           1                    Macon_-_Bibb_County         ORG
6      4           1                             Brightmark         ORG
7      4           1                                  Macon         GPE
8      5           1                     SK_Global_Chemical         ORG
9      5           1                            South_Korea         GPE

可见「Brightmark」仅在第4句被识别为ORG，「NSW Government」完全未被识别。我刚接触spaCy和spacyr，求问问题原因和解决办法。

问题原因 #

1. 模型规模限制：你使用的en_core_web_sm是spaCy的小型预训练模型，其NER组件训练数据有限，对低频实体或上下文信息不足的实体识别能力较弱。小型模型侧重通用场景的基础识别，对商业类实体这类特定领域的识别精度远不如大型模型。
2. 上下文依赖偏差：spaCy的NER依赖上下文判断实体类型。比如第1句中「Brightmark LLC」后接的修饰语未被模型关联为公司标识；第2、3、5句里「Brightmark」的上下文缺乏明确的公司类关键词（如「company」「LLC」），导致模型无法判定其为ORG。
3. 实体边界识别偏差：「NSW government」中模型仅识别了「NSW」为ORG，未包含「government」，这是因为预训练模型对这类复合实体的边界划分存在固有偏差。

解决办法 #

1. 更换更大的预训练模型 #

spaCy提供了中型（en_core_web_md）、大型（en_core_web_lg）和Transformer架构（en_core_web_trf）模型，这些模型的NER组件训练数据更丰富，识别精度更高。更换模型的代码如下：

# 先终止当前模型会话，再下载并初始化大型模型
spacy_finalize()
spacy_download_model("en_core_web_lg")
spacy_initialize(model = "en_core_web_lg")

# 重新解析文本并提取实体
parsed <- spacy_parse(corpus)
entity <- entity_extract(parsed)

2. 添加自定义实体匹配规则 #

利用spaCy的PhraseMatcher定义规则，强制识别特定实体。针对「Brightmark」和「NSW government」的示例代码：

# 获取spaCy的nlp对象
nlp <- spacy_get_nlp()

# 创建短语匹配器
matcher <- spacy$matcher$PhraseMatcher(nlp$vocab, attr="LOWER")

# 添加需要匹配的自定义短语
terms <- list("brightmark", "nsw government")
patterns <- lapply(terms, function(x) nlp(x))
matcher$add("CUSTOM_ORG", patterns)

# 定义自定义实体处理函数
process_custom_entities <- function(doc) {
  matches <- matcher(doc)
  for (match in matches) {
    start <- match[2]
    end <- match[3]
    span <- doc$span(start, end, label="ORG")
    doc$ents <- c(doc$ents, span)
  }
  return(doc)
}

# 应用自定义处理器并提取实体
parsed <- spacy_parse(corpus, processors = list(nlp = process_custom_entities))
entity <- entity_extract(parsed)

3. 结合字典规则增强识别 #

使用quanteda的字典功能，补充实体识别的规则：

# 创建公司实体字典
company_dict <- dictionary(list(ORG = c("Brightmark", "Brightmark LLC", "NSW government")))

# 结合规则提取实体
entity <- entity_extract(parsed, rules = company_dict, overwrite = TRUE)

4. 微调预训练模型 #

如果你的数据属于特定领域，可以收集标注数据微调spaCy的NER模型：

• 手动或用标注工具（如Prodigy）标注你的新闻数据
• 使用spaCy官方训练脚本微调NER组件
• 将微调后的模型导入spacyr使用

内容的提问来源于stack exchange，提问作者aterhorst