为何Curl请求Lowes接口正常但等效Python代码被拦截？

问题情况 #

• 使用Curl请求Lowes产品接口可正常获取JSON响应：

curl --location 'https://www.lowes.com/rnr/r/get-by-product/50413062/pdp/prod?sortMethod=SubmissionTime&sortDirection=desc&offset=0' \
--header 'authority: www.lowes.com' \
--header 'accept: application/json, text/plain, */*' \
--header 'accept-language: en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4' \
--header 'dnt: 1' \
--header 'referer: https://www.lowes.com/pd/KitchenAid-25-8-cu-ft-5-Door-French-Door-Refrigerator-with-Ice-Maker-Stainless-Steel/50413062' \
--header 'sec-ch-ua: "Google Chrome";v="109", "Chromium";v="109", "Not=A?Brand";v="99"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36' \
--header 'Cookie: EPID=OGFhNmEzMWItMTI5Mi00OTk5LWE0MDMtNDE0M2FjZWVhMGYw; _abck=F42771C150B4C119EC8B175032CA9585~-1~YAAQl24/Fxke7uWGAQAA2GM8+wnNceF82+OD7MYSY54sfGjY6sI9gk/3TA4PXZJET2G+qbMoKQblP1E6vLRIN9mNrI7hUixTfWSLLY3M6mtGEAJVhM4QZInOEUYep02CB7YspAAj33IAsz1CjJ6CTVGMkBHaD3xcYTERPO/ciOfY+BBBDxr+3P5xaqF3UYfPCKt7hKF9Mma2Ov/Q3adCanBW3jFQ3pSfaUZJkoJwqLdcydDMT0X/OUfPEBD5o6DlFFeLGNISFMr9GWt3LEI1WgtkQ8JuCEfaPx6wB0OGXt3CT3rSv3FJWOpqQdDkiEld74IHsPhXkxrfD1ujjX6Mhvioab5tcf49CdmtqD9ew+XOjNZGg+fvIx4/qCDzlcTJAQIjOvdiMhowip38mm9gkgCbhh808pb5y3J+mQ==~-1~-1~-1; bm_sz=1A8431CC8D5EB5603F27267A256FB757~YAAQl24/Fxoe7uWGAQAA2GM8+xNq6E5oqP5QkZ1IFVKNETajEDxDQPsGYZPLcy71Ye8/DDeGwZlryGa+D3p18VEw6uIkOsFnLSUbVX50hwSaGaj+KMNvfjWBiRvNSsrk+OFSTKGWpB6e0cbbctbqDFEw1YJDCK5GdAw3ZKSNBxaUnRLQFoFlFpdzBe26MQo6CkIeMx9mq3oNnaajwjEwSyOzbGclasWViYeL0nzNgiKXvfD+MEFOvbtwU/+9ymXhAJUiLUPBQdyCF6aVWspjx9SNaEGR+iuu7jBUjnDcldwGnQ==~3355461~4539959; dbidv2=8aa6a31b-1292-4999-a403-4143aceea0f0; akaalb_prod_dual=1679338684~op=PROD_GCP_EAST_CTRL_DFLT:PROD_DEFAULT_CTRL|~rv=31~m=PROD_DEFAULT_CTRL:0|~os=352fb8a62db4e37e16b221fb4cefd635~id=ece72cac85177577cc3b22d07a88c063; region=central'

• 但使用逻辑等效的Python requests代码请求时，被服务器拦截，返回拒绝访问页面：

import requests

url = "https://www.lowes.com/rnr/r/get-by-product/50413062/pdp/prod?sortMethod=SubmissionTime&sortDirection=desc&offset=0"

payload={}
headers = {
  'authority': 'www.lowes.com',
  'accept': 'application/json, text/plain, */*',
  'accept-language': 'en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4',
  'dnt': '1',
  'referer': 'https://www.lowes.com/pd/KitchenAid-25-8-cu-ft-5-Door-French-Door-Refrigerator-with-Ice-Maker-Stainless-Steel/50413062',
  'sec-ch-ua': '"Google Chrome";v="109", "Chromium";v="109", "Not=A?Brand";v="99"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"macOS"',
  'sec-fetch-dest': 'empty',
  'sec-fetch-mode': 'cors',
  'sec-fetch-site': 'same-origin',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'Cookie': 'EPID=OGFhNmEzMWItMTI5Mi00OTk5LWE0MDMtNDE0M2FjZWVhMGYw; _abck=F42771C150B4C119EC8B175032CA9585~-1~YAAQl24/Fxke7uWGAQAA2GM8+wnNceF82+OD7MYSY54sfGjY6sI9gk/3TA4PXZJET2G+qbMoKQblP1E6vLRIN9mNrI7hUixTfWSLLY3M6mtGEAJVhM4QZInOEUYep02CB7YspAAj33IAsz1CjJ6CTVGMkBHaD3xcYTERPO/ciOfY+BBBDxr+3P5xaqF3UYfPCKt7hKF9Mma2Ov/Q3adCanBW3jFQ3pSfaUZJkoJwqLdcydDMT0X/OUfPEBD5o6DlFFeLGNISFMr9GWt3LEI1WgtkQ8JuCEfaPx6wB0OGXt3CT3rSv3FJWOpqQdDkiEld74IHsPhXkxrfD1ujjX6Mhvioab5tcf49CdmtqD9ew+XOjNZGg+fvIx4/qCDzlcTJAQIjOvdiMhowip38mm9gkgCbhh808pb5y3J+mQ==~-1~-1~-1; bm_sz=1A8431CC8D5EB5603F27267A256FB757~YAAQl24/Fxoe7uWGAQAA2GM8+xNq6E5oqP5QkZ1IFVKNETajEDxDQPsGYZPLcy71Ye8/DDeGwZlryGa+D3p18VEw6uIkOsFnLSUbVX50hwSaGaj+KMNvfjWBiRvNSsrk+OFSTKGWpB6e0cbbctbqDFEw1YJDCK5GdAw3ZKSNBxaUnRLQFoFlFpdzBe26MQo6CkIeMx9mq3oNnaajwjEwSyOzbGclasWViYeL0nzNgiKXvfD+MEFOvbtwU/+9ymXhAJUiLUPBQdyCF6aVWspjx9SNaEGR+iuu7jBUjnDcldwGnQ==~3355461~4539959; dbidv2=8aa6a31b-1292-4999-a403-4143aceea0f0; akaalb_prod_dual=1679338684~op=PROD_GCP_EAST_CTRL_DFLT:PROD_DEFAULT_CTRL|~rv=31~m=PROD_DEFAULT_CTRL:0|~os=352fb8a62db4e37e16b221fb4cefd635~id=ece72cac85177577cc3b22d07a88c063; region=central'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

返回的拒绝访问内容：

'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;www&#46;lowes&#46;com&#47;rnr&#47;r&#47;get&#45;by&#45;product&#47;5013699701&#47;pdp&#47;prod&#63;" on this server.<P>\nReference&#32;&#35;18&#46;b56e3f17&#46;1679223477&#46;1d4c5ef7\n</BODY>\n</HTML>\n'

• 额外情况：将Curl中的User-Agent改为Chrome 108或110版本时，请求也会失效，说明Lowes对UA版本有严格校验。

解决方案 #

关键修改点 #

1. 改用requests.Session()维持会话：先访问产品详情页获取最新的Cookie和会话信息，避免使用过期Cookie。
2. 拆分Cookie为字典传递：直接在headers里传Cookie字符串可能存在编码或格式问题，改用cookies参数传递字典格式的Cookie。
3. 确保请求头格式准确：移除不必要的转义字符，保证请求头和Curl完全一致。
4. 严格固定Chrome 109的User-Agent：不能随意修改版本号，必须和能正常请求的Curl保持一致。

修改后的Python代码 #

import requests

# 初始化会话
session = requests.Session()

# 产品详情页URL
product_url = "https://www.lowes.com/pd/KitchenAid-25-8-cu-ft-5-Door-French-Door-Refrigerator-with-Ice-Maker-Stainless-Steel/50413062"
# 目标接口URL
target_url = "https://www.lowes.com/rnr/r/get-by-product/50413062/pdp/prod?sortMethod=SubmissionTime&sortDirection=desc&offset=0"

# 统一请求头
headers = {
    'authority': 'www.lowes.com',
    'accept': 'application/json, text/plain, */*',
    'accept-language': 'en-US,en;q=0.9,hi;q=0.8,de;q=0.7,ur;q=0.6,pa;q=0.5,es;q=0.4',
    'dnt': '1',
    'referer': product_url,
    'sec-ch-ua': '"Google Chrome";v="109", "Chromium";v="109", "Not=A?Brand";v="99"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

# 先请求产品页，获取会话Cookie
session.get(product_url, headers=headers)

# 若需要补充原始Cookie，可将其拆分为字典添加到session.cookies中（可选，视情况而定）
# cookie_dict = {
#     'EPID': 'OGFhNmEzMWItMTI5Mi00OTk5LWE0MDMtNDE0M2FjZWVhMGYw',
#     # 其他Cookie键值对依次拆分添加
# }
# session.cookies.update(cookie_dict)

# 请求目标接口
response = session.get(target_url, headers=headers)

# 打印响应内容
print(response.text)

补充说明 #

• 如果直接通过会话获取的Cookie仍无法请求成功，可以将Curl中的Cookie拆分为键值对的字典，通过session.cookies.update(cookie_dict)添加到会话中。
• 禁止修改User-Agent的版本号，Lowes的反爬机制会校验UA的版本匹配性。

内容的提问来源于stack exchange，提问作者user8087010