基于pyparsing的谷歌风格搜索解析器容错性改造需求

我使用pyparsing编写了一个解析谷歌风格搜索字符串（如foo AND (bar OR baz)）的小型解析器，完整代码如下：

import pyparsing as pp
from typing import Literal

class TermNode:
  def __init__(self, term_type: Literal["WORD", "PHRASE"], value: "Node"):
    self.term_type = term_type
    self.value = value

  def __repr__(self):
    return f"TermNode({self.term_type}, {self.value})"

class UnaryNode:
  def __init__(self, operator: Literal["NOT"], operand: "Node"):
    self.operator = operator
    self.operand = operand

  def __repr__(self):
    return f"UnaryNode({self.operator}, {self.operand})"

class BinaryNode:
  def __init__(self, operator: Literal["AND", "OR"], left: "Node", right: "Node"):
    self.operator = operator
    self.left = left
    self.right = right

  def __repr__(self):
    return f"BinaryNode({self.operator}, {self.left}, {self.right})"

Node = TermNode | UnaryNode | BinaryNode

not_ = pp.Keyword("NOT")
and_ = pp.Keyword("AND")
or_ = pp.Keyword("OR")
lparen = pp.Literal("(")
rparen = pp.Literal(")")

extra_chars = "_-'",
word = ~(not_ | and_ | or_) + pp.Word(pp.alphanums + pp.alphas8bit + extra_chars).set_parse_action(lambda t: TermNode("WORD", t[0]))
phrase = pp.QuotedString(quoteChar='"').set_parse_action(lambda t: TermNode("PHRASE", t[0]))
term = (phrase | word)

or_expression = pp.Forward()

parens_expression = pp.Forward()
parens_expression <<= (pp.Suppress(lparen) + or_expression + pp.Suppress(rparen)) | term

not_expression = pp.Forward()
not_expression <<= (not_ + not_expression).set_parse_action(lambda t: UnaryNode("NOT", t[1])) | parens_expression

and_expression = pp.Forward()
and_expression <<= (not_expression + and_ + and_expression).set_parse_action(lambda t: BinaryNode("AND", t[0], t[2])) | (not_expression + and_expression).set_parse_action(lambda t: BinaryNode("AND", t[0], t[1])) | not_expression

or_expression <<= (and_expression + or_ + or_expression).set_parse_action(lambda t: BinaryNode("OR", t[0], t[2])) | and_expression

#or_expression.parse_string('', parse_all=True)
or_expression.run_tests("""\
  ###
  # Valid expressions
  ###

  # Word term
  foobar

  # Umlaute in word term
  Gürtel

  # Phrase term
  "foo bar"

  # Special characters in phrase
  "foo!~ bar %"

  # Implicit AND
  foo bar

  # Explicit AND
  foo AND bar

  # Explicit OR
  foo OR bar

  # NOT
  NOT foo

  # Parenthesis
  foo AND (bar OR baz)

  # Complex expression 1
  NOT foo AND ("bar baz" OR qux)

  # Complex expression 2
  foo AND (NOT "bar baz" (moo OR zoo) AND yoo)

  # Complex expression 3
  foo (bar NOT "baz moo") zoo

  ###
  # Invalid expressions
  ###

  # Unary before binary operator
  foo NOT AND bar

  # Invalid redundant operators
  foo AND OR bar

  # Unknown char outside quoted terms
  foo ~ bar

  # Binary operator at start of line
  AND foo

  # Binary operator at start of parens expression
  (AND bar)

  # Binary operator at end of line
  foo AND

  # Binary operator at end of parens expression
  (foo AND)

  # Unary operator at end of line
  foo NOT

  # Unary operator at end of parens expression
  (foo NOT)

  # Unbalanced parens
  ((foo)

  # Unbalanced quotes
  ""foo"
""");

我希望让该解析器具备完全容错性，像谷歌搜索一样忽略错误并尽可能多地完成解析。目前面临选择困惑：是调整解析语法（对我来说难度较高），还是添加预处理步骤使搜索字符串在解析时始终有效（但存在不少边界情况，详见测试用例中的无效表达式）？此外，我尝试过使用pyparsing的search_string替代parse_string，它不会抛出异常，但输出结果往往不符合需求（例如foo AND OR bar的解析结果为[[TermNode(WORD, foo)], [BinaryNode(OR, TermNode(WORD, ND), TermNode(WORD, bar))]]）。

优先选择调整解析语法而非预处理，原因是预处理很难覆盖所有边界情况，而谷歌的容错逻辑本质是解析层的错误恢复，并非单纯的字符串修正。以下是具体的pyparsing调整方案：

1. 为表达式添加容错分支 #

修改递归表达式定义，加入错误处理分支，跳过无效语法结构：

• 针对连续运算符（如AND OR）：在and_expression和or_expression中，添加逻辑识别连续运算符，跳过无效冗余运算符，只保留可形成有效表达式的组合。
• 针对开头/结尾的孤立运算符：定义分支直接跳过无匹配项的运算符，比如开头的AND foo会忽略AND，仅解析foo；结尾的foo AND会忽略AND，仅解析foo。

示例修改and_expression：

and_expression = pp.Forward()
# 基础有效AND组合
and_expr_core = (not_expression + and_ + and_expression).set_parse_action(lambda t: BinaryNode("AND", t[0], t[2]))
# 处理连续运算符：跳过中间无效项，保留首尾有效元素
and_expr_fault_tolerant = (not_expression + pp.OneOrMore(and_ | or_) + not_expression).set_parse_action(
    lambda t: BinaryNode("AND" if "AND" in t else "OR", t[0], t[-1])
)
# 隐式AND组合
and_expr_implicit = (not_expression + and_expression).set_parse_action(lambda t: BinaryNode("AND", t[0], t[1]))
# 合并所有分支，包含容错逻辑
and_expression <<= and_expr_fault_tolerant | and_expr_core | and_expr_implicit | not_expression

2. 处理非法字符与不平衡符号 #

• 非法字符：定义匹配规则，遇到非预期字符（如~）时直接跳过，不中断解析流程。
• 不平衡括号/引号：为不平衡符号添加匹配规则，左括号后无右括号时匹配到行尾；右括号无对应左括号时直接跳过；未闭合的引号自动转为普通文本处理。

示例代码：

# 跳过非法字符
invalid_char = pp.Regex(r'[^\w\s"\(\)ANDORNOT-_\'’]').set_parse_action(lambda: None)
# 处理不平衡左括号
unbalanced_lparen = pp.Suppress(lparen) + pp.SkipTo(rparen | pp.LineEnd()) + pp.Optional(pp.Suppress(rparen))
# 处理不平衡右括号
unbalanced_rparen = rparen.set_parse_action(lambda: None)
# 处理未闭合的引号：转为普通短语
unbalanced_phrase = pp.Regex(r'"[^"]*$').set_parse_action(lambda t: TermNode("PHRASE", t[0].strip('"')))

# 更新term和parens_expression定义，加入容错逻辑
term = pp.SkipTo(phrase | unbalanced_phrase | word | not_ | and_ | or_ | lparen | rparen, include=False) + (phrase | unbalanced_phrase | word)
parens_expression <<= (pp.Suppress(lparen) + or_expression + pp.Suppress(rparen)) | unbalanced_lparen | unbalanced_rparen | term

3. 利用pyparsing错误恢复机制 #

使用set_fail_action为关键表达式设置错误恢复逻辑，解析失败时跳过当前错误部分，继续解析后续内容：

# 为or_expression设置错误恢复，失败时提取后续第一个有效词继续解析
or_expression.set_fail_action(lambda s, l, e: pp.ParseResults([TermNode("WORD", s[l:].split()[0]) if s[l:] else None]))

为何不推荐预处理 #

预处理需要手动覆盖所有边界情况（连续运算符、不平衡符号、非法字符等），逻辑复杂度高且极易遗漏场景。比如处理foo NOT AND bar时，预处理既要识别NOT AND是无效组合，又要区分NOT (AND bar)这种合法场景，逻辑极易出错。而解析层的容错可利用pyparsing递归匹配特性，更精准处理语法错误。

内容的提问来源于stack exchange，提问作者medihack