复制开发者工具POST请求抓取Vietstock表格数据失败求助

问题核心 #

调用KQGDThongKeGiaStockPaging接口时，即便带上从页面提取的__RequestVerificationToken，仍返回there's nothing here，本质是请求缺少会话完整性验证和关键请求头标识。

修正后的代码 #

import requests
from bs4 import BeautifulSoup

def extract_verification_token(html):
    soup = BeautifulSoup(html, 'html.parser')
    token_input = soup.find('input', {'name': '__RequestVerificationToken'})
    return token_input['value'] if token_input else None

# 用Session自动管理会话Cookie，确保请求连贯性
session = requests.Session()

base_url = "https://finance.vietstock.vn/ket-qua-giao-dich/vietnam.aspx?tab=thong-ke-gia&exchange=1&code=-19"
api_url = "https://finance.vietstock.vn/data/KQGDThongKeGiaStockPaging"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Referer": base_url,
    "X-Requested-With": "XMLHttpRequest"  # 新增AJAX请求标识头
}

# 第一步：获取页面及对应会话的验证令牌
response = session.get(base_url, headers=headers)
if response.status_code != 200:
    print(f"获取页面失败：{response.status_code}")
    exit()

verification_token = extract_verification_token(response.text)
if not verification_token:
    print("未找到验证令牌。")
    exit()

# 第二步：构造完整请求负载并发送POST
payload = {
    "__RequestVerificationToken": verification_token,
    "page": 1,
    "pageSize": 20,
    "catID": 1,
    "stockID": -19,
    "fromDate": "2023-12-07",
    "toDate": "2024-01-07"
}

response = session.post(api_url, data=payload, headers=headers)
if response.status_code == 200:
    try:
        data = response.json()
        print("获取数据成功：")
        print(data)
    except ValueError:
        print("返回内容非JSON格式：")
        print(response.text)
else:
    print(f"请求失败：{response.status_code}")
    print(response.text)

关键调整说明 #

• 会话一致性：使用requests.Session()自动保存GET页面时的Cookie，POST请求会自动复用，解决了之前独立请求导致的会话不匹配问题。
• AJAX标识头：添加X-Requested-With: XMLHttpRequest，这是多数网站验证AJAX请求合法性的必要标识。
• Token与Cookie绑定：确保验证令牌和当前会话的Cookie对应，避免跨会话Token无效的问题。

内容的提问来源于stack exchange，提问作者hq_bk