如何用Spacy移除文本中所有ORG实体？解决n-gram难题

我正在开展一个NLP项目，使用Spacy工具。目前已通过Spacy的NER识别出各类实体，希望从原始输入字符串中移除被识别为ORG（组织）的实体。

现有代码：

doc_text = "I'm here with the three of Nikkei Asia's stalwart editors, three Brits in Tokyo. First off, we have Michael Peel, who is executive editor, a journalist from our affiliate, The Financial Times  . He is now in Tokyo but has previously reported from the likes of Brussels, Bangkok, Abu Dhabi  and Lagos. Welcome, Michael.MICHAEL PEEL, EXECUTIVE EDITOR: Welcome Waj. Thank you very much.KHAN: All right. And we have Stephen Foley, our business editor who, like Michael, is on secondment from the FT, where he was deputy U.S. News Editor. Prior to the FT, he was a reporter at The Independent and like Michael, he's a fresh-off-the-boat arrival in Tokyo and has left some pretty big shoes to fill in the New York bureau, where we miss him. Welcome, Stephen.STEPHEN FOLEY, BUSINESS EDITOR: Thanks for having me, Waj.KHAN: Alright, and last but certainly not least, my brother in arms when it comes to cricket commentary across the high seas is Andy Sharp, or deputy editor who joined Nikkei Asia nearly four years ago, after a long stint at Bloomberg  in Tokyo and other esteemed Japanese publications. Welcome, Andy.ANDREW SHARP"

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp(doc_text)
org_stopwords = [ent.text for ent in doc.ents if ent.label_ == 'ORG']

org_stopwords输出：

['The Financial Times  ', 'Abu Dhabi  and Lagos', 'Bloomberg  ']

这些实体属于n-gram，常规字符串拆分方式无法直接移除，恳请提供可行的代码示例解决该问题。

不要直接对原始字符串做替换，而是利用Spacy的Doc对象和实体的位置信息构建新文本，这样能精准跳过ORG实体的所有token，避免字符串匹配带来的误差（比如空格、重复内容等问题）。

代码示例 #

import spacy

# 加载Spacy预训练模型
nlp = spacy.load("en_core_web_sm")

doc_text = "I'm here with the three of Nikkei Asia's stalwart editors, three Brits in Tokyo. First off, we have Michael Peel, who is executive editor, a journalist from our affiliate, The Financial Times  . He is now in Tokyo but has previously reported from the likes of Brussels, Bangkok, Abu Dhabi  and Lagos. Welcome, Michael.MICHAEL PEEL, EXECUTIVE EDITOR: Welcome Waj. Thank you very much.KHAN: All right. And we have Stephen Foley, our business editor who, like Michael, is on secondment from the FT, where he was deputy U.S. News Editor. Prior to the FT, he was a reporter at The Independent and like Michael, he's a fresh-off-the-boat arrival in Tokyo and has left some pretty big shoes to fill in the New York bureau, where we miss him. Welcome, Stephen.STEPHEN FOLEY, BUSINESS EDITOR: Thanks for having me, Waj.KHAN: Alright, and last but certainly not least, my brother in arms when it comes to cricket commentary across the high seas is Andy Sharp, or deputy editor who joined Nikkei Asia nearly four years ago, after a long stint at Bloomberg  in Tokyo and other esteemed Japanese publications. Welcome, Andy.ANDREW SHARP"

# 处理文本得到Spacy Doc对象
doc = nlp(doc_text)

cleaned_content = []
skip_end_idx = -1  # 标记需要跳过的token结束位置

for idx, token in enumerate(doc):
    # 如果当前token在需要跳过的范围内，直接跳过
    if idx < skip_end_idx:
        continue
    # 检查当前token是否属于ORG实体
    if token.ent_type_ == "ORG":
        # 计算该ORG实体的结束token索引，跳过整个实体
        skip_end_idx = token.ent_iob + token.ent_len
        continue
    # 保留当前token和它的原始空格，保证文本格式一致
    cleaned_content.append(token.text + token.whitespace_)

# 拼接成最终清洗后的文本
cleaned_text = ''.join(cleaned_content)
print(cleaned_text)

代码说明 #

1. 精准跳过实体：通过token.ent_iob（实体起始位置）和token.ent_len（实体包含的token数量）计算出ORG实体的结束索引，直接跳过整个实体的所有token，避免逐个判断的繁琐。
2. 保留原始格式：使用token.whitespace_保留文本中的原始空格、标点间距等格式，确保处理后的文本和原文本的排版逻辑一致。
3. 避免替换误差：直接操作Spacy的token对象，不会出现因实体包含特殊字符、重复片段导致的错误替换问题。

内容的提问来源于stack exchange，提问作者Starlord22