如何修复Python请求green-market.by城市切换接口的500/400错误

我需要从green-market.by网站抓取店铺数据。切换城市时，该网站会向select-city接口发送POST请求，但我尝试发送相同的POST请求时出现500错误，发送GET请求则返回400错误，请问该如何解决？

附上我的代码：

from bs4 import BeautifulSoup
import requests

headers = {'Accept': '*/*',
           'Accept-Encoding': 'gzip, deflate, br',
           'Accept-Language': 'en-US,en;q=0.9,ru;q=0.8,ru-RU;q=0.7',
           'Content-Length': '15',
           'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
           'Cookie': '_gcl_au=1.1.1917694700.1697713017; _ym_uid=1671631216368203443; _ym_d=1697713017; tmr_lvid=721ce302ebfa2fc381e0ba8656e20e57; tmr_lvidTS=1671631216752; _tt_enable_cookie=1; _ttp=5j20jXIA4FMkrSig_ZU4TGETjP4; _gid=GA1.2.2125961325.1698045395; _ym_isad=1; _ga=GA1.1.1752787434.1697713017; _ym_visorc=w; tmr_detect=0%7C1698148842875; XSRF-TOKEN=eyJpdiI6IlwvdTI1VGZvMHR1UzRxRHk4Y25iMFBRPT0iLCJ2YWx1ZSI6ImxsTENiOVdOUWlLZDFMNGtcL0poeVVKUHV0NVgydHd0NTIzQzc3Q2k2V3EzZzl0Z3RITzhuV1ZHVm5ZTVdPSlB1IiwibWFjIjoiOTdiOGY3NWNlYTM3YmFlZWJlMTI1YTI4NTdmZDdmZTM2N2U2ZDFkNjgwZjZlNWRkNGYxMDEzZTliY2E2M2Y5NiJ9; laravel_session=eyJpdiI6Imhha1B6c29YODFxSlhWcFQ1QlwvYk5RPT0iLCJ2YWx1ZSI6IkR4XC83ZkpsXC85Mk1SNUE4bHNRMVMyRlcxZ3NQZGRVY3g5S1QrTlk0OENiNlVac0Z1WDdBM0p5U3N0Z212RFdvZCIsIm1hYyI6IjM1ZjEzYTJiZWNmNGVlMGEwNTA4MmNjOTNlZjgzOWVkOTFmNjM2MmUzYjZlNDNjMThlOGE0NDY4YzNlMTBjZDUifQ%3D%3D; _ga_R9H1MPJ25Q=GS1.1.1698146961.6.1.1698149080.9.0.0',
           'Origin': 'https://www.green-market.by',
           'Referer': 'https://www.green-market.by/',
           'Sec-Ch-Ua': '"Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99"',
           'Sec-Ch-Ua-Mobile': '?0',
           'Sec-Ch-Ua-Platform': '"Windows"',
           'Sec-Fetch-Dest': 'empty',
           'Sec-Fetch-Mode': 'cors',
           'Sec-Fetch-Site': 'same-origin',
           'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
           'X-Csrf-Token': 'IelNECpIvQExsCA1dUiyXEJ2Wc0NGDVjaZOOkKwz',
           'X-Requested-With': 'XMLHttpRequest'
          }

url_green = 'https://www.green-market.by/select-city/'
data_green = {'city': '7', 'type': 'web'}
page_green = requests.get(url_green, headers=headers, params=data_green)
soup_green = BeautifulSoup(page_green.content, 'html.parser')

问题排查与修复步骤 #

1. 修正请求方法：网站切换城市使用POST请求，你的代码误用了GET。将requests.get改为requests.post，同时把params=data_green替换为data=data_green——POST参数应放在请求体而非URL中。
2. 删除硬编码的Content-Length：手动设置的Content-Length:15仅对应特定参数长度，参数变化后会导致服务器校验失败。直接移除该header，requests会自动计算正确值。
3. 更新Cookie与CSRF Token：Cookie和X-Csrf-Token具有时效性，你使用的旧值已失效。先请求主页，从响应Cookie和页面meta标签中提取最新的CSRF Token，再用会话保持Cookie一致性。
4. 修复请求头格式：Sec-Ch-Ua等字段中的"是HTML转义字符，需替换为实际双引号，否则服务器无法正确解析这些头信息。

修改后的示例代码 #

from bs4 import BeautifulSoup
import requests

# 初始化会话，自动保持Cookie
session = requests.Session()
home_url = 'https://www.green-market.by/'
home_response = session.get(home_url)
soup = BeautifulSoup(home_response.content, 'html.parser')

# 从页面meta标签提取最新CSRF Token
csrf_token = soup.find('meta', {'name': 'csrf-token'})['content']

# 整理请求头，移除硬编码的Content-Length，修正转义引号
headers = {
    'Accept': '*/*',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9,ru;q=0.8,ru-RU;q=0.7',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'Origin': 'https://www.green-market.by',
    'Referer': 'https://www.green-market.by/',
    'Sec-Ch-Ua': '"Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99"',
    'Sec-Ch-Ua-Mobile': '?0',
    'Sec-Ch-Ua-Platform': '"Windows"',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'X-Csrf-Token': csrf_token,
    'X-Requested-With': 'XMLHttpRequest'
}

# 发送POST请求切换城市
url_green = 'https://www.green-market.by/select-city/'
data_green = {'city': '7', 'type': 'web'}
page_green = session.post(url_green, headers=headers, data=data_green)

# 检查响应状态与内容
print(page_green.status_code)
print(page_green.text)

额外提示 #

如果仍报错，对比浏览器发送的请求头和参数，确保没有遗漏字段；频繁请求可能触发反爬，建议添加请求间隔，避免IP被封禁。

内容的提问来源于stack exchange，提问作者VBB