如何在Antlr4解析PLSQL时获取并打印Token类型？

我使用以下Python代码打印PLSQL源文件中的Token：

from antlr4 import *
from antlr4.tree.Tree import TerminalNodeImpl
from antlr4.tree.Trees import Trees
from PlSqlLexer import PlSqlLexer
from PlSqlParser import PlSqlParser

import sys
import json


def main():
    with open(sys.argv[1], 'r') as file:
        filesrc = file.read()

    lexer = PlSqlLexer(InputStream(filesrc))
    parser = PlSqlParser(CommonTokenStream(lexer))
    tree = parser.sql_script()
    traverse(tree, parser.ruleNames)

def traverse(tree, rule_names, indent = 0):
    tree
    if tree.getText() == "<EOF>":
        return
    elif isinstance(tree, TerminalNodeImpl):
        print("{0}TOKEN='{1}'".format("  " * indent, tree.getText() ))  ## <<< Prints Token
        #print (tree)
        n = 1
    else:
        print("{0}{1}".format("  " * indent, rule_names[tree.getRuleIndex()]))
        for child in tree.children:
            traverse(child, rule_names, indent + 1)

if __name__ == '__main__':
    main()

运行后仅输出Token内容，例如：

TOKEN='CREATE'
  TOKEN='OR'
  TOKEN='REPLACE'
  ...

我希望同时打印Token类型（如procedure start、变量、表等），但尝试用print(json.dumps(tree))和print(json.dumps(parser))查看可用信息时，出现TypeError: Object of type PlSqlLexer is not JSON serializable错误，请教如何实现需求。

要获取并打印Token类型，你需要通过TerminalNodeImpl的getSymbol()方法拿到对应的Token对象，再利用ANTLR生成的Lexer词汇表获取类型的符号名称。修改后的代码如下：

from antlr4 import *
from antlr4.tree.Tree import TerminalNodeImpl
from PlSqlLexer import PlSqlLexer
from PlSqlParser import PlSqlParser

import sys


def main():
    with open(sys.argv[1], 'r') as file:
        filesrc = file.read()

    lexer = PlSqlLexer(InputStream(filesrc))
    parser = PlSqlParser(CommonTokenStream(lexer))
    tree = parser.sql_script()
    traverse(tree, parser.ruleNames)

def traverse(tree, rule_names, indent = 0):
    if tree.getText() == "<EOF>":
        return
    elif isinstance(tree, TerminalNodeImpl):
        token = tree.getSymbol()
        # 获取Token类型的符号名称
        token_type = PlSqlLexer.VOCABULARY.getSymbolicName(token.type)
        # 处理未知类型的情况
        if token_type is None:
            token_type = f"UNKNOWN_TYPE_{token.type}"
        print("{0}TOKEN='{1}' TYPE='{2}'".format("  " * indent, tree.getText(), token_type))
    else:
        print("{0}{1}".format("  " * indent, rule_names[tree.getRuleIndex()]))
        for child in tree.children:
            traverse(child, rule_names, indent + 1)

if __name__ == '__main__':
    main()

关键说明： #

1. 获取Token对象：TerminalNodeImpl的getSymbol()方法返回对应的Token实例，包含类型、位置等元数据。
2. 获取类型名称：通过PlSqlLexer.VOCABULARY.getSymbolicName(token.type)将Token的类型编号转换为可读性强的符号名称（比如CREATE对应类型CREATE，变量名对应IDENTIFIER等）。
3. 处理边界情况：如果遇到未定义的类型，会返回UNKNOWN_TYPE_xxx格式的字符串，避免报错。

修改后运行，输出会类似：

TOKEN='CREATE' TYPE='CREATE'
  TOKEN='OR' TYPE='OR'
  TOKEN='REPLACE' TYPE='REPLACE'
  TOKEN='PROCEDURE' TYPE='PROCEDURE'
  TOKEN='MY_PROC' TYPE='IDENTIFIER'
  ...

内容的提问来源于stack exchange，提问作者Pro West