Python如何移除未闭合标签？BS4与lxml处理痛点求解

BS4和lxml这类DOM解析器会自动补全未闭合的开标签，导致它们在内容末尾被闭合，不符合需求。可以通过直接处理HTML token流的方式实现自定义逻辑，以下是两种可行方案：

方案1：直接移除未配对的开标签 #

通过解析HTML的原始token，维护标签栈跟踪已打开的标签，忽略不匹配的闭合标签，且不为未配对的开标签自动添加闭合标签，相当于移除这类标签的影响。

from bs4 import BeautifulSoup

def remove_unpaired_open_tags(html_content):
    soup = BeautifulSoup(html_content, "html.parser")
    tokenizer = soup.builder.tokenizer
    tokens = list(tokenizer.feed(html_content))
    
    tag_stack = []
    output = []
    
    for token in tokens:
        # 处理开标签（含自闭合标签）
        if token[0] == "StartTag":
            tag_name = token[1]
            is_self_closing = token[2].get("self_closing", False)
            if is_self_closing:
                output.append(token)
            else:
                tag_stack.append(tag_name)
                output.append(token)
        # 处理闭合标签：仅匹配栈顶标签时才出栈并保留
        elif token[0] == "EndTag":
            tag_name = token[1]
            if tag_stack and tag_stack[-1] == tag_name:
                tag_stack.pop()
                output.append(token)
        # 文本、注释等其他内容直接保留
        else:
            output.append(token)
    
    # 用BS4的builder重新构建HTML
    builder = soup.builder
    builder.reset()
    for token in output:
        if token[0] == "StartTag":
            builder.start(token[1], token[2])
        elif token[0] == "EndTag":
            builder.end(token[1])
        elif token[0] == "Comment":
            builder.comment(token[1])
        elif token[0] == "Doctype":
            builder.doctype(token[1])
        else:
            builder.data(token[1])
    return builder.finish()

# 测试示例
codeblock = '<strong>Good</strong> Some text and bad closed strong </strong> Some text and bad open strong PROBLEM HERE <strong> Some text <h2>Some</h2> or <h3>Some</h3> <p>Some Some text <strong>Good2</strong></p>'
print(remove_unpaired_open_tags(codeblock))

运行后，未配对的<strong>开标签不会被自动补全闭合，输出的HTML中不会出现末尾的</strong>。

方案2：在第一个后续标签前闭合未配对的开标签 #

同样基于token流处理，遇到新的开标签时，先闭合当前所有未配对的开标签，再处理新标签，实现未配对标签在第一个同级标签前闭合的效果。

from bs4 import BeautifulSoup

def close_unpaired_before_next_tag(html_content):
    soup = BeautifulSoup(html_content, "html.parser")
    tokenizer = soup.builder.tokenizer
    tokens = list(tokenizer.feed(html_content))
    
    tag_stack = []
    output = []
    
    for token in tokens:
        # 处理开标签：先闭合所有未配对标签，再处理当前标签
        if token[0] == "StartTag":
            tag_name = token[1]
            is_self_closing = token[2].get("self_closing", False)
            # 闭合栈中所有未配对的开标签
            while tag_stack:
                closed_tag = tag_stack.pop()
                output.append(("EndTag", closed_tag, {}))
            if is_self_closing:
                output.append(token)
            else:
                tag_stack.append(tag_name)
                output.append(token)
        # 处理闭合标签：仅匹配栈顶标签时才出栈并保留
        elif token[0] == "EndTag":
            tag_name = token[1]
            if tag_stack and tag_stack[-1] == tag_name:
                tag_stack.pop()
                output.append(token)
        # 文本、注释等其他内容直接保留
        else:
            output.append(token)
    
    # 闭合剩余的未配对标签（可选，根据需求决定是否保留）
    while tag_stack:
        closed_tag = tag_stack.pop()
        output.append(("EndTag", closed_tag, {}))
    
    # 重新构建HTML
    builder = soup.builder
    builder.reset()
    for token in output:
        if token[0] == "StartTag":
            builder.start(token[1], token[2])
        elif token[0] == "EndTag":
            builder.end(token[1])
        elif token[0] == "Comment":
            builder.comment(token[1])
        elif token[0] == "Doctype":
            builder.doctype(token[1])
        else:
            builder.data(token[1])
    return builder.finish()

# 测试示例
codeblock = '<strong>Good</strong> Some text and bad closed strong </strong> Some text and bad open strong PROBLEM HERE <strong> Some text <h2>Some</h2> or <h3>Some</h3> <p>Some Some text <strong>Good2</strong></p>'
print(close_unpaired_before_next_tag(codeblock))

运行后，示例中的未配对<strong>会在<h2>前被闭合，输出符合需求：PROBLEM HERE <strong> Some text </strong><h2>Some</h2>。

内容的提问来源于stack exchange，提问作者Sasha Kucher