使用Beautiful Racket实现词法分析器时ID令牌解析错误求助

问题现象 #

使用Beautiful Racket实现语法解析时，解析器遇到A这类ID令牌时报错：

Encountered unexpected token of type 'ID (value "A") while parsing 'unknown [line=1, column=#f, offset=9]

解析器通过parse-to-datum运行正常，但词法分析器存在匹配缺陷，导致ID无法被正确识别。

待解析示例、语法定义及原词法代码 #

待解析语法示例 #

10 read A 
20 read B 
30 gosub 400 
40 if C = 400 then write C 
50 if C = 0 then goto 1000 
400 C = A + B : return 
$$

语法定义 #

program -> linelist $$ 
linelist -> line linelist | epsilon 
line -> idx stmt linetail* [EOL]
idx -> nonzero_digit digit* 
linetail -> :stmt | epsilon 
stmt -> id = expr | if expr then stmt | read id | write expr | goto idx | gosub idx | return
expr -> id etail | num etail | (expr)
etail -> + expr | - expr | = expr | epsilon
id -> [a-zA-Z]+
num -> numsign digit digit*
numsign -> + | - | epsilon 
nonzero_digit -> 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
digit -> 0 | nonzero_digit

原词法分析器代码 #

#lang br/quicklang
(require brag/support)

(define (make-tokenizer port)
  (port-count-lines! port) ; get line data
  (define (next-token)

    (define-lex-abbrevs
      (lower-letter (:/ "a" "z"))
      (upper-letter (:/ #\A #\Z))
      (digit (:/ "0" "9")))

    (define odai-lexer
      (lexer

       [whitespace (token 'WS lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start) #:skip? #t)]

       ["{" (token 'PROG-START lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["}" (token 'PROG-STOP lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["$" (token 'DOLLAR lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]

       ["read" (token 'READ lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["write" (token 'WRITE lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       [";" (token 'DELIMIT lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]

       ["if" (token 'IF lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["then" (token 'THEN lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]

       ["=" (token 'ASSIGN-OP lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["+" (token 'ADD-OP lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["-" (token 'SUB-OP lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["(" (token 'OPENa-PAREN lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       [")" (token 'CLOSE-PAREN lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]

       ["goto" (token 'GOTO lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["gosub" (token 'GOSUB lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       ["return" (token 'RETURN lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]

       [(:+ (:or lower-letter upper-letter)) (token 'ID lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]
       [(:+ digit) (token 'DIGIT (string->number lexeme) #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]


       [any-char (token 'MISC lexeme #:position (+ (pos lexeme-start)) #:line (line lexeme-start))]))
    (odai-lexer port))
  next-token)

(provide make-tokenizer)

问题根源 #

1. 缺失:令牌匹配：语法中linetail规则包含:stmt，但原词法未定义:的匹配规则，导致:被识别为MISC令牌，打乱解析流程。
2. 未定义EOL令牌：语法中line规则要求[EOL]，但原词法未处理换行符，解析器无法识别行结束标记。
3. 令牌名称拼写错误：OPENa-PAREN应为OPEN-PAREN，避免令牌名称不匹配。
4. 数字令牌命名不匹配：语法中使用num/idx对应数字，原词法用DIGIT，虽不直接导致ID报错，但统一命名更规范。

修改后的词法分析器代码 #

#lang br/quicklang
(require brag/support)

(define (make-tokenizer port)
  (port-count-lines! port)
  (define (next-token)

    (define-lex-abbrevs
      (lower-letter (:/ "a" "z"))
      (upper-letter (:/ #\A #\Z))
      (digit (:/ "0" "9")))

    (define odai-lexer
      (lexer
       ; 跳过空白字符
       [whitespace (token 'WS lexeme #:position (pos lexeme-start) #:line (line lexeme-start) #:skip? #t)]
       ; 换行符生成EOL令牌
       [#\newline (token 'EOL lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ; 程序结束标记
       ["$" (token 'DOLLAR lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ; 关键字
       ["read" (token 'READ lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["write" (token 'WRITE lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["if" (token 'IF lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["then" (token 'THEN lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["goto" (token 'GOTO lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["gosub" (token 'GOSUB lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["return" (token 'RETURN lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ; 运算符与分隔符
       [":" (token 'COLON lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["=" (token 'ASSIGN-OP lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["+" (token 'ADD-OP lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["-" (token 'SUB-OP lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ["(" (token 'OPEN-PAREN lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       [")" (token 'CLOSE-PAREN lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       [";" (token 'DELIMIT lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ; 标识符
       [(:+ (:or lower-letter upper-letter)) (token 'ID lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]
       ; 数字（统一为NUM，语法规则区分行号和普通数字）
       [(:+ digit) (token 'NUM (string->number lexeme) #:position (pos lexeme-start) #:line (line lexeme-start))]
       ; 未匹配字符
       [any-char (token 'MISC lexeme #:position (pos lexeme-start) #:line (line lexeme-start))]))
    (odai-lexer port))
  next-token)

(provide make-tokenizer)

额外语法调整提示 #

确保语法文件中：

• linetail规则里的:对应词法的COLON令牌，即修改为linetail -> COLON stmt | epsilon
• idx和num规则使用NUM令牌，而非原digit（语法文件中需用大写令牌名，因为词法生成的是大写令牌类型）

调整后的关键语法片段示例：

line -> NUM stmt linetail* EOL
linetail -> COLON stmt | epsilon
num -> numsign NUM
idx -> NUM  ; 语法规则会通过上下文保证是非零开头，词法无需额外区分

内容的提问来源于stack exchange，提问作者Flurrshify