Python中使用NLTK正则解析语法标签列表遇ValueError问题求助

问题代码与错误信息 #

你尝试用NLTK的正则解析器判断语句时态，编写了以下代码：

数据预处理代码 #

from nltk import word_tokenize, pos_tag
import nltk

text = "He will have been doing his homework." 

tokenized = word_tokenize(text)
tagged = pos_tag(tokenized)
tags = []
for i in range(len(tagged)):
    t = tagged[i]
    tags.append(t[1])
print(tags)

正则语法规则 #

grammar = r"""
Future_Perfect_Continuous: {<MD><VB><VBN><VBG>}
Future_Continuous:         {<MD><VB><VBG>}
Future_Perfect:            {<MD><VB><VBN>}
Past_Perfect_Continuous:   {<VBD><VBN><VBG>}
Present_Perfect_Continuous:{<VBP|VBZ><VBN><VBG>}
Future_Indefinite:         {<MD><VB>}
Past_Continuous:           {<VBD><VBG>}
Past_Perfect:              {<VBD><VBN>}
Present_Continuous:        {<VBZ|VBP><VBG>}
Present_Perfect:           {<VBZ|VBP><VBN>}
Past_Indefinite:           {<VBD>}
Present_Indefinite:        {<VBZ>|<VBP>}
"""

解析函数 #

def check_grammar(grammar, tags):
    cp = nltk.RegexpParser(grammar)
    result = cp.parse(tags)
    print(result)
    result.draw()
 
check_grammar(grammar, tags)

运行后抛出错误：

Traceback (most recent call last):
  File "/home/samar/Desktop/twitter_tense/main.py", line 35, in <module>
    check_grammar(grammar, tags)
  File "/home/samar/Desktop/twitter_tense/main.py", line 31, in check_grammar
    result = cp.parse(tags)
  File "/home/samar/.local/lib/python3.8/site-packages/nltk/chunk/regexp.py", line 1276, in parse
    chunk_struct = parser.parse(chunk_struct, trace=trace)
  File "/home/samar/.local/lib/python3.8/site-packages/nltk/chunk/regexp.py", line 1083, in parse
    chunkstr = ChunkString(chunk_struct)
  File "/home/samar/.local/lib/python3.8/site-packages/nltk/chunk/regexp.py", line 95, in __init__
    tags = [self._tag(tok) for tok in self._pieces]
  File "/home/samar/.local/lib/python3.8/site-packages/nltk/chunk/regexp.py", line 95, in <listcomp>
    tags = [self._tag(tok) for tok in self._pieces]
  File "/home/samar/.local/lib/python3.8/site-packages/nltk/chunk/regexp.py", line 105, in _tag
    raise ValueError("chunk structures must contain tagged " "tokens or trees")
ValueError: chunk structures must contain tagged tokens or trees

错误原因 #

1. 输入数据格式错误：NLTK的RegexpParser.parse()方法要求输入必须是带标记的token序列（即(单词, 词性标签)的元组列表），但你传入的tags是纯词性标签的字符串列表，不符合方法的输入要求。
2. 语法规则写法错误：Present_Indefinite: {<VBZ>|<VBP>}的写法不符合NLTK chunk语法规范，多个可选标签需要写在同一个尖括号内，否则会被解析为两个独立的规则片段，无法正确匹配单个的VBZ或VBP标签。

修复方案 #

1. 调整输入数据：直接使用pos_tag()返回的tagged元组列表（包含单词和对应标签）作为parse()方法的输入，无需单独提取纯标签列表。
2. 修正语法规则：将Present_Indefinite的规则改为{<VBZ|VBP>}，确保多选标签的写法符合NLTK要求。

修复后的完整代码 #

from nltk import word_tokenize, pos_tag
import nltk

text = "He will have been doing his homework." 

tokenized = word_tokenize(text)
# 直接保留带标记的token元组列表
tagged = pos_tag(tokenized)
print(tagged)

# 修正语法规则中的Present_Indefinite写法
grammar = r"""
Future_Perfect_Continuous: {<MD><VB><VBN><VBG>}
Future_Continuous:         {<MD><VB><VBG>}
Future_Perfect:            {<MD><VB><VBN>}
Past_Perfect_Continuous:   {<VBD><VBN><VBG>}
Present_Perfect_Continuous:{<VBP|VBZ><VBN><VBG>}
Future_Indefinite:         {<MD><VB>}
Past_Continuous:           {<VBD><VBG>}
Past_Perfect:              {<VBD><VBN>}
Present_Continuous:        {<VBZ|VBP><VBG>}
Present_Perfect:           {<VBZ|VBP><VBN>}
Past_Indefinite:           {<VBD>}
Present_Indefinite:        {<VBZ|VBP>}
"""

def check_grammar(grammar, tagged_tokens):
    cp = nltk.RegexpParser(grammar)
    result = cp.parse(tagged_tokens)
    print(result)
    result.draw()
 
# 传入带标记的token列表
check_grammar(grammar, tagged)

运行后就能正确解析时态结构，弹出可视化窗口展示解析结果。

内容的提问来源于stack exchange，提问作者Samar Pratap Singh