使用Python Requests进行AJAX网页爬取无法获取表格数据问题排查

安全阿岳

2026-9-30

问题原因及修复方法

你遇到的问题核心是请求体格式不符合接口要求，具体修改点如下：

原代码中使用data参数传递字典类型的请求载荷，requests会默认将其序列化为表单格式，而该API要求接收JSON格式的请求体
你可以直接使用requests内置的json参数传递载荷，无需手动序列化和添加Content-Type头，框架会自动处理

修正后的代码

import requests

url = 'https://www.assetmanagement.hsbc.de/api/v1/nav/funds'
payload = {"appliedFilters":[[{"active":True,"id":"Yes"}]],"paging":{"fundsPerPage":-1,"currentPage":1},"view":"Documents","searchTerm":[],"selectedValues":[],"pageInformation":{"country":"DE","language":"DE","investorType":"INST","tokenIssue":{"url":"/api/v1/token/issue"},"dataUrl":{"url":"/api/v1/nav/funds","id":"e0FFNDg5MTJELUFEMzEtNEQ5RC04MzA4LTdBQzZERTgyQTc4Rn0="},"shareClassUrl":{"url":"/api/v1/nav/shareclass","id":"ezUxODdjODJiLWY1YmItNDIzOC1hM2Y0LWY5NzZlY2JmMTU3OX0="},"filterUrl":{"url":"/api/v1/nav/filters","id":"ezRFREYxQTU3LTVENkYtNDBDRC1CMjJDLTQ0NDc4Nzc1NTlFQn0="},"presentationUrl":{"url":"/api/v1/nav/presentation","id":"e0E1NEZDODZGLUE5MDctNDUzQi04RTYyLTIxNDNBMEM1MEVGQ30="},"liveDataUrl":{"id":"ezlEMjA2MDk5LUNCRTItNENGMy1BRThBLUM0RTMwMEIzMjlDQ30="},"fundDetailPageUrl":"/de/institutional-investors/fund-centre","forceHttps":True}}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"}

# 这里把data改成json参数
r = requests.post(url, headers=headers, json=payload)
# 先判断请求状态
if r.status_code == 200:
    print(r.json())
else:
    print(f"请求失败，状态码：{r.status_code}，返回内容：{r.text}")