XPath解析网易云音乐榜单因特殊字符致内容不全的解决方法

初始尝试的Python代码如下：

import requests
from lxml import etree

url = "https://music.163.com/discover/toplist?id=19723756"
headers = {
    'User-Agent': "PostmanRuntime/7.15.2",
    }
response = requests.request("GET", url, headers=headers)

'''
"<"、"&"
'''
r = etree.HTML(response.text)

l = r.xpath("//textarea[@id='song-list-pre-data']/text()")

print(l)

执行上述代码后，获取到的列表l内容末尾出现截断，示例截断内容为：lLevel":"exhigh","pl":320000},"djid":0,"fee":0,"album":{"id":158052587,"name":"Sakana~( ˵>ㅿㅿ\n']，因特殊字符导致匹配信息不全。

可解决该问题的Python代码如下：

import requests
from bs4 import BeautifulSoup

url = "https://music.163.com/discover/toplist?id=3779629"
headers = {
    'user-agent': "PostmanRuntime/7.15.2"
    }
response = requests.request("GET", url, headers=headers)

soup = BeautifulSoup(response.text, "html.parser")
textarea = soup.find('textarea', attrs={'id': 'song-list-pre-data'}).get_text()

print(textarea)

解决思路：使用BeautifulSoup进行解析，不依赖lxml，改用Python内置的html.parser解析器，可正确处理包含特殊字符的文本内容，避免截断问题。

内容的提问来源于stack exchange，提问作者rui0908