Curl调用Best Buy评论API成功但Python Requests失败求因

我尝试调用Best Buy网站的API获取商品评论，终端执行以下Curl命令时可正常收到服务器响应：

curl --location 'https://www.bestbuy.com/ugc/v2/reviews?page=2&pageSize=8&sku=6509757&sort=BEST_REVIEW&variant=A' \
--header 'authority: www.bestbuy.com' \
--header 'accept: */*' \
--header 'accept-language: en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4' \
--header 'dnt: 1' \
--header 'referer: https://www.bestbuy.com/site/doctor-strange-in-the-multiverse-of-madness-steelbookdigital-copy-4k-ultra-hd-blu-ray-blu-ray-2022/6509757.p?skuId=6509757' \
--header 'sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36' \
--header 'x-client-id: ratings-and-reviews-user-generated-content-ratings-and-reviews-v1'

但使用Python requests库发起相同API调用时却始终失败，所用代码如下：

import requests

url = "https://www.bestbuy.com/ugc/v2/reviews?page=2&pageSize=8&sku=6509757&sort=BEST_REVIEW&variant=A"

payload={}
headers = {
  'authority': 'www.bestbuy.com',
  'accept': '*/*',
  'accept-language': 'en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4',
  'dnt': '1',
  'referer': 'https://www.bestbuy.com/site/doctor-strange-in-the-multiverse-of-madness-steelbookdigital-copy-4k-ultra-hd-blu-ray-blu-ray-2022/6509757.p?skuId=6509757',
  'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"macOS"',
  'sec-fetch-dest': 'empty',
  'sec-fetch-mode': 'cors',
  'sec-fetch-site': 'same-origin',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'x-client-id': 'ratings-and-reviews-user-generated-content-ratings-and-reviews-v1'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

问题原因 #

• 头部字符串转义错误：Python代码中sec-ch-ua、sec-ch-ua-platform等头部的值使用了HTML转义符"，而Curl中实际传递的是原生双引号。Requests会将这些转义符原样发送给服务器，导致服务器无法正确解析这些标识字段，触发验证失败。
• GET请求冗余参数：GET请求不需要传递data=payload参数，即使payload是空字典，Requests仍可能额外添加Content-Type等无关头部，与Curl的纯GET请求格式不一致，干扰服务器的请求校验。

修正后的代码 #

import requests

url = "https://www.bestbuy.com/ugc/v2/reviews?page=2&pageSize=8&sku=6509757&sort=BEST_REVIEW&variant=A"

headers = {
    'authority': 'www.bestbuy.com',
    'accept': '*/*',
    'accept-language': 'en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4',
    'dnt': '1',
    'referer': 'https://www.bestbuy.com/site/doctor-strange-in-the-multiverse-of-madness-steelbookdigital-copy-4k-ultra-hd-blu-ray-blu-ray-2022/6509757.p?skuId=6509757',
    'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    'x-client-id': 'ratings-and-reviews-user-generated-content-ratings-and-reviews-v1'
}

# GET请求无需data参数，直接发起请求
response = requests.get(url, headers=headers)

print(response.text)

内容的提问来源于stack exchange，提问作者user8087010