ANTLR词法分析器问题：避免浮点数规则误匹配V语言范围运算符

Hey there, I've dealt with similar lexer ambiguity issues before, so let's break down how to fix this problem where your FloatNumber fragment is incorrectly matching parts of the range expression 0..10 (like 0. and .10) instead of letting .. be recognized as a separate punctuation token.

The Root of the Problem #

Your current FloatNumber rule doesn't account for context: when a decimal point is adjacent to another decimal point (like in ..), those shouldn't be part of a floating-point number. Even though 0. is a valid float in V, when it's immediately followed by another ., we need to prioritize the range operator over splitting it into a float plus a stray dot.

Solution: Use Contextual Assertions & Lexer Priority #

We can adjust the FloatNumber rule with negative lookahead and negative lookbehind assertions to prevent it from matching parts of the range operator. Plus, we'll make sure the range operator rule takes precedence.

Updated Lexer Rules #

fragment FloatNumber 
    : ( Digit+ ('.' Digit* )? ([eE] [+-]? Digit+)? 
      | Digit* '.' Digit+ ([eE] [+-]? Digit+)? )
      // Ensure no trailing dot (avoids matching `0.` in `0..10`)
      (?! \. )
      // Ensure no leading dot (avoids matching `.10` in `0..10`)
      (?<! \. )
    ;

fragment Digit : [0-9] ;

// Define the range operator FIRST - lexers prioritize longer matches first
RangeOperator : '..' ;

How This Works #

• Negative Lookahead (?! \. ): This checks that after matching a float, there isn't another dot right after it. So 0. in 0..10 won't be flagged as a float because the next character is ., which fails the assertion.
• Negative Lookbehind (?<! \. ): This checks that before matching a float (like .1), there isn't a dot immediately preceding it. So .10 in 0..10 won't be matched as a float since the previous character is ..
• Lexer Priority: Lexers always favor longer token matches first. By defining RangeOperator : '..' before the float rules, the lexer will grab .. as a single token instead of splitting it into two dots that could combine with surrounding digits.

Test Cases to Validate #

• Valid floats like 1. or .1 will match correctly—no adjacent dots, so the assertions pass.
• The range expression 0..10 will be split into 0 (integer), .. (RangeOperator), and 10 (integer) instead of being misinterpreted as float fragments.

Fallback for Tools Without Lookbehind Support #

If your lexer tool doesn't support negative lookbehind (like older ANTLR versions), you can rely solely on the longest match priority:

// Define range operator first to ensure it's matched before float fragments
RangeOperator : '..' ;

fragment FloatNumber 
    : Digit+ ('.' Digit* )? ([eE] [+-]? Digit+)? 
    | '.' Digit+ ([eE] [+-]? Digit+)? 
    ;

Since .. is a longer token than a single ., the lexer will pick it first, leaving no stray dots to combine with the surrounding numbers into invalid float matches.

内容的提问来源于stack exchange，提问作者Thomas S.