Python使用suds MessagePlugin与lxml仅对XML中description字段做HTML解码

问题背景 #

从接口获取商品信息，使用suds的MessagePlugin作为过滤器解析返回数据，返回的SOAP XML报文片段如下：

<env:Envelope xmlns:env='http://schemas.xmlsoap.org/soap/envelope/'><env:Header></env:Header><env:Body><prod:getProductsResponse xmlns:prod='https://product.individual.ns.listinsgapi.aa.com'><return><ackCode>success</ackCode><responseTime>13/09/2021 09:47:34</responseTime><timeElapsed>211 ms</timeElapsed><productCount>199</productCount><products><product><productId>01201801947</productId><product><categoryCode>cn1g</categoryCode><storeCategoryId>0</storeCategoryId><title>Morphy Richards Sensörlü çöp kutusu, 30 litre, yuvarlak, siyah paslanmaz çelik</title><specs><spec required="false" value="Standart Çöp Kovası" name="Ürün Tipi"/><spec required="false" value="Montajsız" name="Montaj Tipi"/><spec required="false" value="Sensörlü Kapak" name="Kapak Tipi"/><spec required="false" value="26 lt-30 lt" name="İç Hacim"/><spec required="false" value="Çelik" name="Malzeme"/><spec required="false" value="Sıfır" name="Durum"/></specs><photos><photo photoId="0"><url>https://mcdn301.gi1ttigidliyor.net/622080/620801947_0.jpg</url></photo><photo photoId="1"><url>https://mcdn011.gittigidliyor.net/620380/62081101947_1.jpg</url></photo><photo photoId="2"><url>https://mcdn021.gittigidliyor.net/620180/6210801947_2.jpg</url></photo><photo photoId="3"><url>https://mcdn201.gittigidliyor.net/620850/6208013947_3.jpg</url></photo><photo photoId="4"><url>https://mcdn301.gittigidliyor.net/623080/6208101947_4.jpg</url></photo><photo photoId="5"><url>https://mcdn01.gittigidiyor.net/62080/620801947_5.jpg</url></photo><photo photoId="6"><url>https://mcdn01.gittigidiyor.net/62080/620801947_6.jpg</url></photo></photos><pageTemplate>4</pageTemplate><description>&lt;body&gt;
 &lt;ul class=&quot;a-unordered-list a-vertical a-spacing-mini&quot; style=&quot;padding-right: 0px; padding-left: 0px; box-sizing: border-box; margin: 0px 0px 0px 18px; color: rgb(17, 17, 17); font-family: &quot;&gt; 
  &lt;li style=&quot;box-sizing: border-box; list-style: disc; overflow-wrap: break-word; margin: 0px;&quot;&gt;&amp;nbsp; &lt;h2 style=&quot;box-sizing: border-box; padding: 0px 0px 4px; margin: 3px 0px 7px; text-rendering: optimizelegibility; line-height: 32px; font-family: &quot;&gt;Ürün Bilgileri&lt;/h2&gt; &lt;span style=&quot;background-color:rgb(255, 255, 255); box-sizing:border-box; color:rgb(15, 17, 17); font-family:amazon ember,arial,sans-serif; font-size:14px&quot;&gt;Renk:&lt;strong style=&quot;box-sizing:border-box; font-weight:700&quot;&gt;Paslanmaz Çelik&lt;/strong&gt;&lt;/span&gt; 
   &lt;div class=&quot;a-row a-spacing-top-base&quot; style=&quot;box-sizing: border-box; width: 1213px; color: rgb(15, 17, 17); font-family: &quot;&gt; 
    &lt;div class=&quot;a-column a-span6&quot; style=&quot;box-sizing: border-box; margin-right: 24.25px; float: left; min-height: 1px; overflow: visible; width: 593.734px;&quot;&gt; 
     &lt;div class=&quot;a-row a-spacing-base&quot; style=&quot;box-sizing: border-box; width: 593.734px; margin-bottom: 12px !important;&quot;&gt; 
      &lt;div class=&quot;a-row a-expander-container a-expander-extend-container&quot; style=&quot;box-sizing: border-box; width: 593.734px;&quot;&gt; 
       &lt;div class=&quot;a-row&quot; style=&quot;box-sizing: border-box; width: 593.734px;&quot;&gt;

问题说明 #

仅需要对报文中的description字段做HTML解码：因为部分商品的description段HTML标签未完整解析，会触发解析错误，错误示例如：

0979c08d37cd.CR0,0,2000,2000_PT0_SX220.jpg
style=-webkit-tap-highlight-color:transparent; border:none;
box-sizing:border-box; display:block; margin:0px auto; max-width:100%;
padding:0px; vertical-align:top/p /div /th /tr /tbody /table /div /div
/div /div /div /div /div /div /div /body

已尝试的方案 #

方案1 #

class UnicodeFilter(MessagePlugin):
    def received(self, context):
        from lxml import etree
        from io import BytesIO
        parser = etree.XMLParser(recover=True)
        request_string = context.reply.decode("utf-8")
        replaced_string = request_string.replace("&gt;", ">").replace("&lt;", "<")
        byte_rep_string = str.encode(replaced_string)
      
        doc = etree.parse(BytesIO(byte_rep_string), parser)
        byte_str_doc = etree.tostring(doc)
        context.reply = byte_str_doc

存在问题：运行无报错，但description内的HTML标签仍未修复，无实际效果。

方案2 #

class UnicodeFilter(MessagePlugin):
    def received(self, context):
        from lxml import etree
        from io import BytesIO
        import html
        parser = etree.XMLParser(recover=True) # Initialize the parser
        request_string = context.reply.decode("utf-8") # Converting incoming data byte to string
        html_decoded = html.unescape(request_string) # Html decoding to the data
        byte_rep_string = str.encode(html_decoded) # Converting the data from string to byte
      
        doc = etree.parse(BytesIO(byte_rep_string), parser)
        byte_str_doc = etree.tostring(doc)
        context.reply = byte_str_doc

存在问题：触发TypeNotFound: Type not found: 'body'报错。

核心需求 #

• 使用lxml解析返回报文，解决"not well-formed (invalid token)"非法字符报错（该问题已解决）；
• 仅对XML报文中的description字段单独做HTML解码，修复标签解析问题。

问题原因分析 #

• 方案1仅替换了<和>两种实体，没有处理"、&等其他HTML实体，因此没有实际效果。
• 方案2对整个XML报文做HTML解码，导致description内的HTML标签（比如<body>）被识别为XML元素节点，破坏了原有SOAP报文的结构，suds解析时找不到对应类型就会抛出TypeNotFound错误。

正确实现代码 #

class UnicodeFilter(MessagePlugin):
    def received(self, context):
        from lxml import etree
        from io import BytesIO
        import html

        # 1. 用recover模式直接解析原始XML报文，解决非法字符问题
        parser = etree.XMLParser(recover=True)
        doc = etree.parse(BytesIO(context.reply), parser)

        # 2. 声明命名空间，匹配报文中的前缀
        ns = {
            'env': 'http://schemas.xmlsoap.org/soap/envelope/',
            'prod': 'https://product.individual.ns.listinsgapi.aa.com'
        }
        # 查找所有description节点
        description_nodes = doc.xpath('//description', namespaces=ns)

        for node in description_nodes:
            if node.text:
                # 仅对description的文本内容做HTML解码
                decoded_desc = html.unescape(node.text)
                # 赋值回节点时lxml会自动处理XML转义，保证整个XML结构合法
                node.text = decoded_desc

        # 3. 序列化处理后的XML替换原始响应
        context.reply = etree.tostring(doc, encoding='utf-8', xml_declaration=True)

内容的提问来源于stack exchange，提问作者bufferoverflow