ANTLR4语法错误：为何a[1+2]无法被正常识别？

问题描述 #

我正在开发一门新语言，使用ANTLR4编写语法（代码如下）。尝试解析数组访问语句a[1+2]时，出现错误提示：mismatched input '1+2' expecting {NUMBER_INT, VARIABLE}。但写成a[1+ 2]（运算符两侧加空格）时却能正常工作，不清楚空格为何会产生如此大的差异，也不知道该修改语法的哪部分来解决这个问题。

原语法代码 #

/* Casual has to be the filename */
grammar Grammar;

/* non-terminals start with lowercase */

prog: (comment|declaration|NEWLINE|SPACE|definition|statement)*EOF;

comment:
    '(*' (ANYCHAR|VARIABLE|SPACE|NEWLINE|number|operator)* '*)'
;

declaration:
    vname_type '('args_def?')' SPACE* ';'
;

vname_type:
    (VARIABLE) SPACE* ':' SPACE* type SPACE* refinement? /* Nomes das funções não podem começar com números */
;

type:
    (DOUBLE|INT|BOOLEAN|FLOAT|STRING)
;

refinement:
    'where' SPACE* (VARIABLE) SPACE* operator SPACE* (number|string_lit|TRUE|FALSE)
;

number:
    (NUMBER_INT|NUMBER_FLOAT)
;

string_lit:
    '"' (ANYCHAR|number|operator|SPACE)* '"'
;

statement:
    (return_statement|expression|value|if_statement|while_statement|arrays)
;

value:
    vname_type '=' SPACE* (number|string_lit|VARIABLE) SPACE* ';'
;

arrays:
    (VARIABLE '['position']' ';'
    |'get_array()['position']' ';'
    )
;

position:
    pos
    |pos SPACE* (MATH_OPERATOR) SPACE* pos
    |pos SPACE* (MATH_OPERATOR) SPACE* pos SPACE* (MATH_OPERATOR) SPACE* position
;
pos:
    (NUMBER_INT|VARIABLE)
;

expression:
    (function_call
    | (number|VARIABLE|ANYCHAR|SPACE)*';')
;

function_call:
    VARIABLE'('args_value? ')' ';'
;

return_statement:
    'return' SPACE* (VARIABLE|number) SPACE* ';'
;

definition:
    (function)
;

function:
    vname_type SPACE* '('SPACE* args_def? SPACE*')' SPACE*
    '{'
    (statement|SPACE)*
    '}'
;

if_statement:
    'if' SPACE* boolean_expression SPACE* '{'
    statement*
    '}' else_statement?
;

else_statement:
    'else' '{' statement* '}'
;

while_statement:
    'while' boolean_expression '{' statement* '}'
;

boolean_expression:
    (conditions_values
    | conditions_values operator conditions_values
    | (conditions_values operator conditions_values operator boolean_expression)
    )
;

conditions_values:
    VARIABLE
    |TRUE
    |FALSE
    |number
;

args_def:
    vname_type SPACE* (',' vname_type SPACE*)*
;

args_value:
    (number|string_lit|TRUE|FALSE) (',' (number|string_lit|TRUE|FALSE))*
;

operator:
    (MATH_OPERATOR | BOOLEAN_OPERATOR)
;

/* terminals start with uppercase, and can be defined using regular expressions. */

DOUBLE: 'Double';
INT: 'Int';
BOOLEAN: 'Boolean';
FLOAT: 'Float';
STRING: 'String';
TRUE: 'true';
FALSE: 'false';
NUMBER_INT: [0-9_]+; /* Underscore can be in any position */
NUMBER_FLOAT: ('.'[0-9]+|[0-9]+.[0-9]+);
MATH_OPERATOR: '+' | '-' | '*' | '/' | '%';
BOOLEAN_OPERATOR: '&&' | '||' | '==' | '!=' | '>=' | '<=' | '<' | '>' ;
NEWLINE : [\r\n]+ -> skip;
SPACE: (' '|'\t') -> skip;
VARIABLE: [a-zA-Z_][a-zA-Z0-9_]*;
ANYCHAR: (.)+?;

问题原因 #

1. 词法规则冲突：ANYCHAR规则(.)+?是优先级极高的通配符，当输入1+2时，词法分析器会把整个序列匹配成一个ANYCHAR token，而非拆分出NUMBER_INT、MATH_OPERATOR、NUMBER_INT。加空格时，SPACE会被跳过，三个token能被正确识别，语法规则position自然可以匹配。
2. NUMBER_FLOAT正则错误：原规则里的.未转义，会匹配任意字符，导致错误的浮点数识别。
3. position规则强制空格：当前position要求运算符两侧必须有SPACE*，无空格时无法匹配拆分后的token序列。

修复方案 #

1. 移除ANYCHAR规则 #

ANYCHAR会干扰所有其他词法规则的匹配，建议直接删除。如果需要处理注释内的任意字符，可以修改comment规则，用更精确的匹配替代ANYCHAR：

comment:
    '(*' ( ~[*] | '*' ~[)] )* '*)'
;

2. 修正NUMBER_FLOAT规则 #

NUMBER_FLOAT: ('.'[0-9]+|[0-9]+'\.'[0-9]+);

3. 简化position规则，支持无空格表达式 #

将表达式逻辑抽成独立规则，同时支持带空格和无空格的写法：

position: expr;
expr:
    pos
    | expr (SPACE* MATH_OPERATOR SPACE*) expr
;
pos: NUMBER_INT | VARIABLE;

这样既支持1+2也支持1+ 2，还能处理嵌套表达式（比如1+2*3）。

4. 优化expression规则 #

原expression规则过于宽泛，修改为更严谨的匹配：

expression:
    function_call
    | expr ';'
;

修改后的完整语法 #

/* Casual has to be the filename */
grammar Grammar;

/* non-terminals start with lowercase */

prog: (comment|declaration|definition|statement)* EOF;

comment:
    '(*' ( ~[*] | '*' ~[)] )* '*)'
;

declaration:
    vname_type '(' args_def? ')' SPACE* ';'
;

vname_type:
    VARIABLE SPACE* ':' SPACE* type SPACE* refinement? /* Nomes das funções não podem começar com números */
;

type:
    DOUBLE | INT | BOOLEAN | FLOAT | STRING
;

refinement:
    'where' SPACE* VARIABLE SPACE* operator SPACE* (number|string_lit|TRUE|FALSE)
;

number:
    NUMBER_INT | NUMBER_FLOAT
;

string_lit:
    '"' ( ~["] )* '"'
;

statement:
    return_statement | expression | value | if_statement | while_statement | arrays
;

value:
    vname_type '=' SPACE* (number|string_lit|VARIABLE) SPACE* ';'
;

arrays:
    (VARIABLE '[' position ']' ';'
    |'get_array()[' position ']' ';'
    )
;

position: expr;
expr:
    pos
    | expr (SPACE* MATH_OPERATOR SPACE*) expr
;
pos: NUMBER_INT | VARIABLE;

expression:
    function_call
    | expr ';'
;

function_call:
    VARIABLE '(' args_value? ')' ';'
;

return_statement:
    'return' SPACE* (VARIABLE|number) SPACE* ';'
;

definition:
    function
;

function:
    vname_type SPACE* '(' SPACE* args_def? SPACE* ')' SPACE*
    '{'
    (statement)*
    '}'
;

if_statement:
    'if' SPACE* boolean_expression SPACE* '{'
    statement*
    '}' else_statement?
;

else_statement:
    'else' '{' statement* '}'
;

while_statement:
    'while' boolean_expression '{' statement* '}'
;

boolean_expression:
    conditions_values
    | conditions_values operator conditions_values
    | conditions_values operator conditions_values operator boolean_expression
;

conditions_values:
    VARIABLE
    | TRUE
    | FALSE
    | number
;

args_def:
    vname_type SPACE* (',' vname_type SPACE*)*
;

args_value:
    (number|string_lit|TRUE|FALSE) (',' (number|string_lit|TRUE|FALSE))*
;

operator:
    MATH_OPERATOR | BOOLEAN_OPERATOR
;

/* terminals start with uppercase, and can be defined using regular expressions. */

DOUBLE: 'Double';
INT: 'Int';
BOOLEAN: 'Boolean';
FLOAT: 'Float';
STRING: 'String';
TRUE: 'true';
FALSE: 'false';
NUMBER_INT: [0-9_]+; /* Underscore can be in any position */
NUMBER_FLOAT: ('.'[0-9]+|[0-9]+'\.'[0-9]+);
MATH_OPERATOR: '+' | '-' | '*' | '/' | '%';
BOOLEAN_OPERATOR: '&&' | '||' | '==' | '!=' | '>=' | '<=' | '<' | '>' ;
NEWLINE : [\r\n]+ -> skip;
SPACE: (' '|'\t') -> skip;
VARIABLE: [a-zA-Z_][a-zA-Z0-9_]*;

内容的提问来源于stack exchange，提问作者rfpg1