Logos词法分析器中Match语句未捕获CloseParen Token问题

问题场景 #

在实现mal项目的词法分析阶段，使用Logos crate处理输入(1 2 3)时，read_list函数的match语句始终无法进入Ok(Token::CloseParen)分支，反而会进入Ok(_)分支调用read_from，最终触发panic。

最小复现代码 #

use logos::{Logos, Lexer};
use std::fmt;

#[derive(Debug, PartialEq)]
enum MalVal {
    Number(i64),
    List(Vec<MalVal>),
}

impl fmt::Display for MalVal {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            MalVal::Number(n) => write!(f, "{}", n),
            MalVal::List(l) => write!(f, "({})", l.iter().map(|v| v.to_string()).collect::<Vec<_>>().join(" ")),
        }
    }
}

#[derive(Debug, PartialEq, Logos)]
enum Token {
    #[regex(r"\d+", |lex| lex.slice().parse())]
    Number(i64),

    #[token("(")]
    OpenParen,

    #[token(")")]
    CloseParen,
}

fn read_from(lex: &mut Lexer<'_, Token>) -> Result<MalVal, String> {
    match lex.next() {
        Some(Ok(Token::Number(n))) => Ok(MalVal::Number(n)),
        Some(Ok(Token::OpenParen)) => read_list(lex),
        Some(Ok(t)) => Err(format!("Unexpected token: {:?}", t)),
        Some(Err(e)) => Err(format!("Lex error: {}", e)),
        None => Err("Unexpected end of input".to_string()),
    }
}

fn read_list(lex: &mut Lexer<'_, Token>) -> Result<MalVal, String> {
    let mut list = Vec::new();
    loop {
        match lex.next() {
            Some(Ok(Token::CloseParen)) => break,
            Some(Ok(t)) => list.push(read_from(lex)?),
            Some(Err(e)) => return Err(format!("Lex error: {}", e)),
            None => return Err("Unclosed list".to_string()),
        }
    }
    Ok(MalVal::List(list))
}

fn main() {
    let input = "(1 2 3)";
    let mut lex = Token::lexer(input);
    let result = read_from(&mut lex);
    println!("Result: {:?}", result);
}

报错回溯信息 #

thread 'main' panicked at src/main.rs:36:25: Unexpected token: CloseParen
stack backtrace:
   0: rust_begin_unwind
             at /rustc/abcabcabc/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/abcabcabc/library/core/src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/abcabcabc/library/core/src/result.rs:1651:5
   3: mal_logos::read_from
             at src/main.rs:36:25
   4: mal_logos::read_list
             at src/main.rs:47:38
   5: mal_logos::read_from
             at src/main.rs:34:29
   6: mal_logos::main
             at src/main.rs:59:22
   7: core::ops::function::FnOnce::call_once
             at /rustc/abcabcabc/library/core/src/ops/function.rs:250:5

Cargo.toml配置 #

[package]
name = "mal_logos"
version = "0.1.0"
edition = "2021"

[dependencies]
logos = "0.12.0"

问题原因 #

核心问题在于read_list的迭代器处理逻辑错误：
当解析到3之后，lex.next()返回Ok(CloseParen)，但read_list的match进入Ok(_)分支，调用read_from(lex)——而read_from会再次调用lex.next()，此时CloseParen被传递到read_from中，read_from无法处理该Token直接panic，导致read_list的CloseParen分支永远无法被触发。

解决方案 #

1. 修正read_list的逻辑，直接处理当前Token #

新增read_from_token函数处理已获取的Token，避免重复调用lex.next()导致Token流错位：

fn read_from_token(token: Token, lex: &mut Lexer<'_, Token>) -> Result<MalVal, String> {
    match token {
        Token::Number(n) => Ok(MalVal::Number(n)),
        Token::OpenParen => read_list(lex),
        t => Err(format!("Unexpected token: {:?}", t)),
    }
}

fn read_list(lex: &mut Lexer<'_, Token>) -> Result<MalVal, String> {
    let mut list = Vec::new();
    loop {
        match lex.next() {
            Some(Ok(Token::CloseParen)) => break,
            Some(Ok(t)) => list.push(read_from_token(t, lex)?),
            Some(Err(e)) => return Err(format!("Lex error: {}", e)),
            None => return Err("Unclosed list".to_string()),
        }
    }
    Ok(MalVal::List(list))
}

2. 调整Token定义顺序（可选但推荐） #

将固定符号类Token（如括号）定义在正则匹配类Token之前，避免Logos优先匹配正则规则导致符号被误解析：

#[derive(Debug, PartialEq, Logos)]
enum Token {
    #[token("(")]
    OpenParen,

    #[token(")")]
    CloseParen,

    #[regex(r"\d+", |lex| lex.slice().parse())]
    Number(i64),
}

3. 调试Token流（排查问题时使用） #

打印所有解析出的Token，确认输入字符的识别结果是否符合预期：

fn main() {
    let input = "(1 2 3)";
    let mut lex = Token::lexer(input);
    println!("Token stream:");
    while let Some(token) = lex.next() {
        println!("{:?}", token);
    }
    // 重新创建Lexer进行解析
    let mut lex = Token::lexer(input);
    let result = read_from(&mut lex);
    println!("Result: {:?}", result);
}

修正后的运行结果 #

Token stream:
Ok(OpenParen)
Ok(Number(1))
Ok(Number(2))
Ok(Number(3))
Ok(CloseParen)
Result: Ok(List([Number(1), Number(2), Number(3)]))

内容的提问来源于stack exchange，提问作者iiqof