使用rply解析多行代码时遇意外令牌错误的解决方法

The problem here is that your current grammar only knows how to parse a single say statement—it has no rule for handling multiple consecutive statements. When the parser finishes the first say("yo");, it doesn't know what to do with the next SAY token, hence the "Ran into a SAY where it wasn't expected" error. The shift/reduce warning is also tied to this incomplete grammar structure.

Here's how to fix it step by step:

1. Add a Top-Level "Program" Rule #

We need a root rule that represents an entire program, which can be either a single statement or a sequence of multiple statements. This tells the parser exactly how to handle back-to-back commands.

2. Refactor Statements into a Reusable Rule #

Rename your existing say production to be part of a generic statement rule—this makes it easy to add other statement types (like variable assignments) later on.

Modified Parser Code #

Here's the updated version of your Parser class with these fixes:

from rply import ParserGenerator
from ast import Int, Sum, Sub, Say, String

class Parser():
    def __init__(self):
        self.pg = ParserGenerator(
            ['INTEGER', 'SAY', 'OPEN_PAREN', 'CLOSE_PAREN', 'SEMI_COLON', 'SUM', 'SUB', 'STRING']
        )

    def parse(self):
        # Top-level rule: a program is one or more statements
        @self.pg.production('program : statement')
        @self.pg.production('program : program statement')
        def program(p):
            if len(p) == 1:
                # Single statement: return as a list
                return [p[0]]
            else:
                # Multiple statements: append to the existing list
                p[0].append(p[1])
                return p[0]

        # Generic statement rule (add other statement types here later)
        @self.pg.production('statement : SAY OPEN_PAREN expression CLOSE_PAREN SEMI_COLON')
        def say(p):
            return Say(p[2])

        # Your existing expression rules stay unchanged
        @self.pg.production('expression : expression SUM expression')
        @self.pg.production('expression : expression SUB expression')
        def expression(p):
            left = p[0]
            right = p[2]
            operator = p[1]
            if operator.gettokentype() == 'SUM':
                return Sum(left, right)
            elif operator.gettokentype() == 'SUB':
                return Sub(left, right)

        @self.pg.production('expression : INTEGER')
        def int(p):
            return Int(p[0].value)

        @self.pg.production('expression : STRING')
        def string(p):
            return String(p[0].value)

        @self.pg.error
        def error_handler(token):
            raise ValueError("Ran into a %s where it wasn't expected" % token.gettokentype())

    def get_parser(self):
        return self.pg.build()

3. Update Execution Logic #

Since the parser now returns a list of statements instead of a single one, adjust your main code to loop through and evaluate each statement:

# Example main code adjustment
parser = Parser()
parser.parse()
rply_parser = parser.get_parser()

tokens = ... # Your token stream here
statements = rply_parser.parse(tokens)
for stmt in statements:
    stmt.eval()

Why This Works #

• The program rule uses recursion to explicitly define that a valid program can be a single statement, or a program plus another statement—this handles any number of consecutive commands.
• Grouping say under a statement rule creates a flexible structure that scales if you add more statement types later.
• The shift/reduce warning should disappear (or be minimized) because the grammar now has a clear hierarchy for parsing sequences.

Give this a try—your multi-line say("yo"); say("yoyo"); commands should execute without errors now!

内容的提问来源于stack exchange，提问作者George Walker