使用lxml.etree解析同一HTML时不同XPath导致编码差异问题

使用lxml.etree解析同一HTML内容时，通过不同XPath获取元素后，用etree.tostring()序列化得到的文本出现编码差异——部分非ASCII字符（如芬兰语中的ä）在一种XPath结果中保留原字符，另一种则被转为HTML实体（如ä）。

可复现代码 #

from lxml import etree
from io import StringIO

def parse_html(html_, xpath):
    html_parser = etree.HTMLParser()
    listing_page_parsed = etree.parse(StringIO(html_), html_parser).xpath(xpath)
    listing_page_parsed = [etree.tostring(item, encoding='unicode') for item in listing_page_parsed]
    return listing_page_parsed

listing_page = """<div class="positions-container" style="position:_|_relative;_|_height:_|_4309.41px;" a="a"> <div class="item-container_|_software_development_and_architecture_|_remote_work_|_helsinki_|_jyväskylä_|_lahti_|_oulu_|_rauma_|_tampere_|_turku_|_vaasa" style="position:_|_absolute;_|_left:_|_0px;_|_top:_|_0px;" a="a"> </div> </div>"""

xpath_1 = "//*"
xpath_2 = '//div[@class="positions-container"]//div[contains(@class,"item-container")]'


result_1 = parse_html(listing_page, xpath_1)[0]
print("Parsing using xpath_1:", f"...{result_1[190:220]}...")
result_2 = parse_html(listing_page, xpath_2)[0]
print("Parsing using xpath_2:", f"...{result_2[85:125]}...")

输出结果 #

Parsing using xpath_1: ...lsinki_|_jyväskylä_|_lahti_|_o...
Parsing using xpath_2: ...lsinki_|_jyväskylä_|_lahti_|_o...

原因分析 #

问题出在etree.tostring()的序列化规则上：

• 当通过xpath_1 = "//*"获取的是HTMLParser自动补全后的根节点（<html>标签），序列化整个HTML文档时，lxml默认使用HTML序列化规则，会保留属性中的原非ASCII字符。
• 当通过xpath_2获取单个<div>节点时，etree.tostring()默认使用XML序列化规则，会将非ASCII字符转换为HTML实体（如ä转为ä）。

解决方法 #

在调用etree.tostring()时显式指定method='html'，强制使用HTML序列化规则，这样无论序列化整个文档还是单个节点，都会保留原字符：

修改后的代码：

from lxml import etree
from io import StringIO

def parse_html(html_, xpath):
    html_parser = etree.HTMLParser()
    listing_page_parsed = etree.parse(StringIO(html_), html_parser).xpath(xpath)
    # 新增method='html'参数，强制使用HTML序列化规则
    listing_page_parsed = [etree.tostring(item, encoding='unicode', method='html') for item in listing_page_parsed]
    return listing_page_parsed

listing_page = """<div class="positions-container" style="position:_|_relative;_|_height:_|_4309.41px;" a="a"> <div class="item-container_|_software_development_and_architecture_|_remote_work_|_helsinki_|_jyväskylä_|_lahti_|_oulu_|_rauma_|_tampere_|_turku_|_vaasa" style="position:_|_absolute;_|_left:_|_0px;_|_top:_|_0px;" a="a"> </div> </div>"""

xpath_1 = "//*"
xpath_2 = '//div[@class="positions-container"]//div[contains(@class,"item-container")]'


result_1 = parse_html(listing_page, xpath_1)[0]
print("Parsing using xpath_1:", f"...{result_1[190:220]}...")
result_2 = parse_html(listing_page, xpath_2)[0]
print("Parsing using xpath_2:", f"...{result_2[85:125]}...")

修改后的输出：

Parsing using xpath_1: ...lsinki_|_jyväskylä_|_lahti_|_o...
Parsing using xpath_2: ...lsinki_|_jyväskylä_|_lahti_|_o...

内容的提问来源于stack exchange，提问作者Minions