如何基于自定义分词列表，用Spacy或其他NLP工具生成CONLL格式的POS/NER/依存分析结果？

Absolutely! You can absolutely use spaCy with your custom token list instead of relying on its built-in tokenizer—this is a common use case, and spaCy has explicit support for it. You can retain your custom tokenization while still running POS tagging, NER, and dependency parsing. Here's how to modify your existing function to work with a token list input:

Step 1: Manually Create a spaCy Doc Object #

The core trick here is bypassing spaCy's tokenizer by creating a Doc object directly from your custom tokens. The Doc is spaCy's central data structure, and all downstream components (parser, NER) operate on it.

Step 2: Run Downstream Pipeline Components #

Once you've created the Doc with your tokens, you just need to run the parser (for dependency parsing and POS tags) and NER components manually, since we skipped the tokenizer step.

Modified Function Code #

import spacy
# Use a modern, maintained model instead of the old 'en' alias
nlp = spacy.load('en_core_web_sm')

def toConll_from_tokens(token_list, nlp):
    # Create Doc from custom tokens, skipping tokenization
    doc = spacy.tokens.Doc(nlp.vocab, words=token_list)
    
    # Retrieve the parser and NER components from the pipeline
    parser = nlp.get_pipe("parser")
    ner = nlp.get_pipe("ner")
    
    # Run the components on our custom Doc
    doc = parser(doc)
    doc = ner(doc)
    
    # Generate CONLL format output (same logic as your original function)
    block = []
    for i, word in enumerate(doc):
        # Calculate head index for CONLL (1-based, root is 0)
        head_idx = 0 if word.head == word else word.head.i + 1
        
        line = [
            str(i+1),
            str(word),
            word.lemma_,
            word.tag_,
            word.ent_type_,
            str(head_idx),
            word.dep_
        ]
        block.append(line)
    return block

# Test with your example token list
custom_tokens = ["Donald", "Trump", "is", "the", "new", "president", "of", "the", "United", "States", "of", "America"]
conll_output = toConll_from_tokens(custom_tokens, nlp)

Key Notes: #

• POS Tags: The parser component automatically handles POS tagging (since spaCy's parser relies on POS features), so you don't need to run a separate POS pipe.
• Model Choice: The old en model is deprecated—using en_core_web_sm (or larger models like en_core_web_md) ensures you get up-to-date functionality and better performance.
• Index Calculation: Since our custom Doc uses 0-based indexing for tokens, we convert the head index to 1-based (per CONLL standards) by adding 1, with the root node marked as 0.

Alternative Tools (If Needed) #

If you're open to other NLP libraries, Stanford CoreNLP also supports pre-tokenized input (you can enable whitespace tokenization to use your custom tokens). However, spaCy's approach is more streamlined for Python workflows and integrates seamlessly with your existing CONLL formatting logic.

内容的提问来源于stack exchange，提问作者dada