添加Content-Type为application/json时Etherscan接口返回500及HTML异常

问题描述 #

我尝试从指定地址的NFT转账页面获取100行数据，该页面通过AJAX请求https://etherscan.io/nft-transfers.aspx/GetTableData_NftTransfers接口返回JSON格式的100行数据。但使用Python的httpx框架请求时遇到两个问题：

• 不携带Content-Type头时，返回仅含25行数据的HTML响应；
• 添加Content-Type: application/json头时，返回500状态码。

以下是原请求代码、请求头及负载信息：

原请求代码 #

nft_transactions = "https://etherscan.io/nft-transfers?a=0x90f37809cccf13705a1f5e9fca645ba3fd58773c"
nft_transaction_post_url = "https://etherscan.io/nft-transfers.aspx/GetTableData_NftTransfers"

with Client(http2=True) as client:
    resp = client.get(nft_transactions, headers=HEADERS, timeout=30)
    print(resp.status_code)
    HEADERS["Referer"] = "https://etherscan.io/nft-transfers?a=0x90f37809cccf13705a1f5e9fca645ba3fd58773c"
    HEADERS["Accept"] = "application/json, text/javascript, */*; q=0.01"
    HEADERS["Content-Type"] = "application/json"
    HEADERS["X-Requested-With"] = "XMLHttpRequest"
    HEADERS["Cookie"] = "ASP.NET_SessionId=pluwldlu2g1prywcul3xm54k; bitmedia_fid=eyJmaWQiOiIzMzVkMWI3MTFhZmQ2NDQ2YzZkY2U5Y2ViYjRmMjJjOSIsImZpZG5vdWEiOiJhYjliNmRiYjk2Y2Q2ZmZmMzBlNjAwMWM0NDFmOTMxNyJ9; etherscan_cookieconsent=True; displaymode=light; _gid=GA1.2.1296145474.1687081165; etherscan_switch_age_datetime=Age; _ga_T1JC9RNQXV=deleted; __cflb=02DiuFnsSsHWYH8WqVXbZzkeTrZ6gtmGUCT1KWiuzFuy6; cf_clearance=uv8ynqXUbgnHxXnMjQv_V_oWrdyogkfcQTbXWmtsq2M-1687510221-0-150; _gat_gtag_UA_46998878_6=1; __cf_bm=yRJObmuJxZnXbIzZCgX.BmSUQ_xLstUEHCrheJZ5CGg-1687529771-0-Af9WCdcIeQIKrGBpEBsw5WlJ875g9lbHKD93cEdNntyt8KfJBqlWQTD4V8WjmUsN5A==; _ga_T1JC9RNQXV=GS1.1.1687529769.78.1.1687529785.0.0.0; _ga=GA1.2.1028829820.1685819037"
    HEADERS["Sec-Fetch-Mode"] = "cors"
    HEADERS["Sec-Fetch-Dest"] = "empty"
    del HEADERS["Sec-Fetch-User"]
    resp = client.post(nft_transaction_post_url, data=payload, headers=HEADERS)
    print(resp.status_code)

原请求头与负载 #

HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,"
              "application/signed-exchange;v=b3;q=0.7",
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
    "Cache-Control": "max-age=0",
    "Sec-Ch-Ua": '"Chromium";v="113", "Not-A.Brand";v="24"',
    "Sec-Ch-Ua-Platform": "Windows",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "same-origin",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 "
                  "Safari/537.36",
    "Origin": "https://etherscan.io"
}

payload = {
  "dataTableModel": {
    "draw": 2,
    "columns": [
      {"data": "preview", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "txhash", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "txMethod", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "dt", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "_from", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "arrow", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "_to", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "type", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "tokenAddress", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}}
    ],
    "order": [],
    "start": 0,
    "length": 100,
    "search": {"value": "", "regex": False},
    "Ext": "0x90f37809cccf13705a1f5e9fca645ba3fd58773c"
  }
}

问题分析 #

1. Payload传递方式错误：设置Content-Type: application/json后，用data=payload会将字典以表单格式发送，后端无法解析JSON格式的参数，导致500错误。
2. 请求头冗余冲突：原HEADERS包含针对页面导航的字段（如Cache-Control、Upgrade-Insecure-Requests），这些字段会干扰AJAX请求的处理逻辑。
3. 硬编码Cookie失效：手动设置的Cookie可能已过期，且无法适配会话变化，导致请求被后端拦截。

修复方案 #

修复后的代码 #

import httpx

nft_transactions_url = "https://etherscan.io/nft-transfers?a=0x90f37809cccf13705a1f5e9fca645ba3fd58773c"
api_url = "https://etherscan.io/nft-transfers.aspx/GetTableData_NftTransfers"

# 精简后的AJAX请求头
BASE_HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "Accept": "application/json, text/javascript, */*; q=0.01",
    "X-Requested-With": "XMLHttpRequest",
    "Referer": nft_transactions_url,
    "Origin": "https://etherscan.io",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Dest": "empty",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8"
}

payload = {
  "dataTableModel": {
    "draw": 2,
    "columns": [
      {"data": "preview", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "txhash", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "txMethod", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "dt", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "_from", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "arrow", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "_to", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "type", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}},
      {"data": "tokenAddress", "name": "", "searchable": True, "orderable": False, "search": {"value": "", "regex": False}}
    ],
    "order": [],
    "start": 0,
    "length": 100,
    "search": {"value": "", "regex": False},
    "Ext": "0x90f37809cccf13705a1f5e9fca645ba3fd58773c"
  }
}

with httpx.Client(http2=True, follow_redirects=True) as client:
    # 先请求页面获取会话Cookie
    get_resp = client.get(nft_transactions_url, headers=BASE_HEADERS)
    get_resp.raise_for_status()
    
    # 使用json参数自动序列化并设置Content-Type
    post_resp = client.post(api_url, json=payload, headers=BASE_HEADERS)
    print(f"响应状态码: {post_resp.status_code}")
    
    if post_resp.status_code == 200:
        response_data = post_resp.json()
        print(f"成功获取{len(response_data['data'])}条NFT转账数据")
        # 此处可添加数据处理逻辑

关键修改点 #

• 使用json=payload：httpx会自动将字典序列化为JSON字符串，并设置Content-Type: application/json，符合后端接口的参数要求。
• 精简请求头：移除了Cache-Control、Upgrade-Insecure-Requests等冗余字段，只保留AJAX请求必需的头信息。
• 自动管理Cookie：通过httpx的Client会话自动保存和传递GET请求获取的Cookie，避免硬编码Cookie过期问题。
• 移除手动Cookie设置：不再手动添加Cookie，让会话自动处理，提升请求可靠性。

内容的提问来源于stack exchange，提问作者ABHISHEK TIWARI