如何像浏览器一样处理HTML提取文本的空白与<p><br>标签？

我需要从XHTML表格里提取纯文本，要求保留浏览器会渲染的换行（比如<p>和<br />带来的换行），但要去掉原始XML文件里的冗余空白（那些浏览器不会显示的换行、多余空格）。原始表格单元格里有很多浏览器不渲染的空白，同时存在<p>和<br />这类会产生换行的标签。

示例1：多段落单元格 #

原始单元格代码 #

<td>
  <p class="TableText10pts"><span style="FONT-SIZE: 11pt; COLOR: black">INTERPOLATION QUEUE FULL 
  </span><span style="FONT-SIZE: 11pt"></span></p>
  <p class="TableText10pts"><span style="FONT-SIZE: 11pt; COLOR: black">In 
  Interpolated position motion mode the set-point buffer is full. The last 
  received set-point is not interpolated.</span><span style="FONT-SIZE: 11pt"></span></p></td>

期望提取结果 #

INTERPOLATION QUEUE FULL
In Interpolated position motion mode the set-point buffer is full. The last received set-point is not interpolated.

（段落间带空行的版本也可以接受）

示例2：含加粗标签的单元格 #

原始单元格代码 #

<td style="BORDER-TOP: medium none; HEIGHT: 13.5pt; BORDER-RIGHT: red 1pt solid; WIDTH: 205.55pt; BACKGROUND: white; BORDER-BOTTOM: red 1pt solid; PADDING-BOTTOM: 0pt; PADDING-TOP: 0pt; PADDING-LEFT: 5.4pt; BORDER-LEFT: red 1pt solid; PADDING-RIGHT: 5.4pt" valign="top" width="274">
  <p class="TableText10pts"><span style="FONT-SIZE: 11pt; COLOR: black">Motor 
  stuck - the motor is powered but is not moving according to the definition 
  of <b>CL[2]</b> and <b>CL[3].</b></span><span style="FONT-SIZE: 11pt"></span></p></td>

期望提取结果 #

Motor stuck - the motor is powered but is not moving according to the definition of CL[2] and CL[3].

遇到的问题 #

• 使用BeautifulSoup的get_text(separator=' ', strip=True)时，原始XML里的非渲染空白会被保留；
• 自定义的BeautifulSoup逻辑仍残留非渲染换行；
• 默认设置的Html2Text会忽略<p>和<br>标签，还会插入多余换行，即使设置body_width=0仍丢失布局；
• 尝试text = [" ".join(p.getText(strip=True).replace("\n", "").split()) for p in soup]时，会丢失<b>这类标签前后的空格，导致文本粘连：

Motor stuck - the motor is powered but is not moving according to the definition ofCL[2]andCL[3].

解决方案 #

要实现和浏览器一致的空白处理，核心逻辑是：清理文本节点内的冗余空白，给块级/换行标签添加对应换行，同时保证行内标签前后保留空格避免粘连。用BeautifulSoup可以这样实现：

from bs4 import BeautifulSoup, NavigableString, Tag

def clean_text_node(text):
    # 清理单个文本节点的冗余空白：替换连续空白为单个空格，去除首尾空白
    return ' '.join(text.strip().split())

def extract_browser_like_text(element):
    result = []
    for child in element.children:
        if isinstance(child, NavigableString):
            cleaned = clean_text_node(child)
            if cleaned:
                result.append(cleaned)
        elif isinstance(child, Tag):
            # 处理块级标签：<p>提取文本后添加换行（要空行就改成'\n\n'）
            if child.name == 'p':
                inner_text = extract_browser_like_text(child)
                if inner_text:
                    result.append(inner_text)
                    result.append('\n')
            # 处理换行标签：<br>直接添加换行
            elif child.name == 'br':
                result.append('\n')
            # 处理行内标签：递归提取文本，前后加空格防止粘连
            else:
                inner_text = extract_browser_like_text(child)
                if inner_text:
                    result.append(' ')
                    result.append(inner_text)
                    result.append(' ')
    # 合并后二次清理：合并连续空格，同时保留换行结构
    final_text = ''.join(result).strip()
    # 清理每行内的连续空格
    final_text = '\n'.join([' '.join(line.split()) for line in final_text.split('\n')])
    return final_text

# 测试示例2
html = '''<td style="BORDER-TOP: medium none; HEIGHT: 13.5pt; BORDER-RIGHT: red 1pt solid; WIDTH: 205.55pt; BACKGROUND: white; BORDER-BOTTOM: red 1pt solid; PADDING-BOTTOM: 0pt; PADDING-TOP: 0pt; PADDING-LEFT: 5.4pt; BORDER-LEFT: red 1pt solid; PADDING-RIGHT: 5.4pt" valign="top" width="274">
  <p class="TableText10pts"><span style="FONT-SIZE: 11pt; COLOR: black">Motor 
  stuck - the motor is powered but is not moving according to the definition 
  of <b>CL[2]</b> and <b>CL[3].</b></span><span style="FONT-SIZE: 11pt"></span></p></td>'''

soup = BeautifulSoup(html, 'lxml')
td_element = soup.find('td')
print(extract_browser_like_text(td_element))

代码说明 #

• clean_text_node：处理单个文本节点的冗余空白，统一替换为单个空格；
• extract_browser_like_text：递归遍历元素子节点，针对不同标签类型做处理：
  • 文本节点直接清理后加入结果；
  • <p>标签提取内部文本后添加换行，可调整为\n\n实现段落间空行；
  • <br>标签直接添加换行；
  • 行内标签（如<b>、<span>）递归提取文本，前后加空格避免文本粘连；
• 最后对整体结果做二次清理，确保连续空格被合并，同时保留换行结构。

测试示例1会输出带段落换行的文本，测试示例2则会保留<b>标签前后的空格，避免文本粘连。

内容的提问来源于stack exchange，提问作者Hydrargyrum