Lex中invalid token处理：扫描到无效令牌/字符时应返回特殊错误码还是调用EXIT_FAILURE？

Great question about handling invalid tokens in Lex—let’s break down the two common approaches and when to use each, with practical code examples to illustrate.

1. Returning a Special Error Token/Code #

This approach is ideal if you need graceful error handling—like reporting specific error details, checking for multiple errors in one run, or allowing users to fix input and retry. Instead of crashing immediately, you return a dedicated error token to your parser (e.g., Yacc/Bison) so it can handle the issue intentionally.

Here’s a Lex snippet demonstrating this:

%{
#include "y.tab.h"  // Assumes you're using Yacc, which defines INVALID_TOKEN
%}

%%
[a-zA-Z][a-zA-Z0-9]*    { return IDENTIFIER; }
[0-9]+                  { return NUMBER; }
"+"                     { return PLUS; }
// Add other valid token rules here...

.                       { return INVALID_TOKEN; }  // Catch-all for invalid characters
%%

Then in your Yacc parser, you can define error handling logic to report the issue clearly:

%token IDENTIFIER NUMBER PLUS INVALID_TOKEN
%%
expr: expr PLUS expr | IDENTIFIER | NUMBER
    ;
%%

int yyerror(char *s) {
    fprintf(stderr, "Error: %s at invalid token '%s' (line %d)\n", s, yytext, yylineno);
    return 0;
}

This lets you pinpoint where the invalid token occurred, show the problematic character, and even continue scanning for other errors before exiting—way more user-friendly for compilers or complex tools.

2. Immediate Termination with exit(EXIT_FAILURE) #

If you’re building a lightweight, single-purpose tool (like a quick input validator or simple scanner) where invalid input means the job can’t proceed at all, terminating immediately is the simplest solution. You skip passing the error to a parser and just crash with a clear message.

Example Lex code for this approach:

%%
[a-zA-Z][a-zA-Z0-9]*    { /* Process valid identifier */ }
[0-9]+                  { /* Process valid number */ }
// Add other valid rules here...

.                       { 
    fprintf(stderr, "Fatal error: Invalid character '%c' found at line %d\n", yytext[0], yylineno);
    exit(EXIT_FAILURE); 
}
%%

This keeps your code concise and avoids unnecessary error recovery logic when there’s no point in continuing execution.

Key Takeaways #

• Use special error tokens/codes when you need detailed error reporting, multiple error detection, or graceful recovery.
• Use immediate exit for simple tools where invalid input makes further processing meaningless.

内容的提问来源于stack exchange，提问作者August Karlstrom