使用Python PLY构建Lexer报错'Can't build lexer'求助

错误信息 #

ERROR: Rule 't_TIMES' defined for an unspecified token TIMES
ERROR: Rule 't_DIVIDE' defined for an unspecified token DIVIDE
Traceback (most recent call last):
  File "...\Lexer.py", line 24, in <module>
    lexer = lex.lex()
            ^^^^^^^^^
  File "...\lex.py", line 910, in lex
    raise SyntaxError("Can't build lexer")
SyntaxError: Can't build lexer

问题原因及修复方案 #

1. 令牌未声明 #

PLY要求所有通过t_<TOKEN_NAME>定义的令牌规则，必须提前在tokens列表中声明。代码中定义了t_TIMES和t_DIVIDE，但tokens列表里缺少这两个令牌，导致lexer构建失败。

修复：在tokens列表中添加对应令牌：

tokens = [
    "INT",
    "ID",
    "PLUS",
    "MINUS",
    "TIMES",  # 新增
    "DIVIDE", # 新增
    "EOF",
]

2. 正则表达式元字符未转义 #

+、*是正则表达式的特殊元字符，直接使用会引发语法错误，需要用反斜杠\转义。

修复：修改对应令牌的正则表达式：

t_PLUS = r"\+"   # 原r"+"改为r"\+"
t_TIMES = r"\*"  # 原r"*"改为r"\*"

3. t_error函数未跳过非法字符 #

当前t_error仅打印错误，但未跳过非法字符，会导致lexer卡在错误位置无法继续解析。

修复：在t_error中添加跳过逻辑：

def t_error(t):
    print(f"Illegal character '{t.lexer.lexeme}'", file=sys.stderr)
    t.lexer.skip(1)  # 跳过当前非法字符

4. INT令牌类型转换（可选但必要） #

原代码中t_INT匹配的数字是字符串类型，无法直接进行数学运算，需要添加类型转换。

修复：将t_INT改为函数形式，转换为整数：

def t_INT(t):
    r"\d+"
    t.value = int(t.value)
    return t

修复后的完整代码 #

import ply.lex as lex
import ply.yacc as yacc
import sys

tokens = [
    "INT",
    "ID",
    "PLUS",
    "MINUS",
    "TIMES",
    "DIVIDE",
    "EOF",
]

def t_INT(t):
    r"\d+"
    t.value = int(t.value)
    return t

t_ID = r"[a-zA-Z_][a-zA-Z0-9_]*"
t_PLUS = r"\+"
t_MINUS = r"-"
t_TIMES = r"\*"
t_DIVIDE = r"/"

def t_error(t):
    print(f"Illegal character '{t.lexer.lexeme}'", file=sys.stderr)
    t.lexer.skip(1)

lexer = lex.lex()

def p_expression(p):
    """expression : INT
                 | ID
                 | expression PLUS expression
                 | expression MINUS expression
                 | expression TIMES expression
                 | expression DIVIDE expression"""
    if len(p) == 2:
        p[0] = p[1]
    else:
        if p[2] == "+":
            p[0] = p[1] + p[3]
        elif p[2] == "-":
            p[0] = p[1] - p[3]
        elif p[2] == "*":
            p[0] = p[1] * p[3]
        elif p[2] == "/":
            p[0] = p[1] / p[3]

parser = yacc.yacc()

def test(text):
    try:
        result = parser.parse(text)
        if result is not None:
            print(result)
        else:
            print("Empty expression")
    except yacc.YaccError:
        print("Error parsing input")

if __name__ == "__main__":
    test("123")          # 输出123
    test("hello")        # 输出hello
    test("123 + 456")    # 输出579
    test("123 - 456")    # 输出-333
    test("123 * 456")    # 输出56088
    test("123 / 456")    # 输出0.26973684210526316

内容的提问来源于stack exchange，提问作者FoxH Gaming