网页爬取遇403错误求助：Postman正常Python请求失败

尝试爬取cef.co.uk的商品价格，Postman中可正常获取数据，但将Postman的curl转为Python代码后，请求返回403错误。已复制浏览器Header信息到代码中，问题仍未解决。以下是相关信息：

浏览器请求头（Chrome DevTools复制） #

:authority: www.cef.co.uk
:method: GET
:path:/basket_line/validation?id=5313783&quantity=1
:scheme: https
accept:application/json, text/plain, /
accept-encoding: gzip, deflate, br,zstd
accept-language: en-GB,en-US;q=0.9,en;q=0.8
cookie:OptanonAlertBoxClosed=2024-12-04T09:35:36.756Z;_gcl_au=1.1.528955057.1733304937; _ga=GA1.1.616359135.1733304937; optimizelyEndUserId=oeu1733304937165r0.26566098801333826;_fbp=fb.2.1733304937302.51198626763862526; _mkto_trk=id:050-FJS-973&token:_mch-cef.co.uk-1733304937323-18536; optimizelySession=0; _session_id=63a31b8ec56f16c9a2a51875193bfca3;FPID=FPID2.3.TKDQTYfrZFVO5i%2Bzb9Ssvyif4wVkkfnIn4eQYBKyBy8%3D.1733304937;_gtmeec=e30%3D; userStatus=Returning; newUser=false; FPLC=TiOmU1B90kYTSWbKZ%2Fq6MxP8lQejKiMexU0YAsz5A1X4h%2Bc9W1sx3hksA9w1kEZ4BOyuKXkQgk4ZuqW38WGWESH%2BEKrlqOQV7oOX47K%2FxDmlJIA1Toe3Aw%2Fe%2FPpM7Q%3D%3D;__cf_bm=tCiJ56LdtivjrzvaarlrdItIF9OvC7J5dds.mrQRZpo-1736949915-1.0.1.1-9kSRN5OrAi5mQHLFtjce3R5Xa765jMEHiKEsbGkoKCkjENJXANE17.gPrc6qR_ZVToKQ0TKDEFeW84ts7zkpdA;cto_bundle=v9hJBV9kblB4aE5ycHE0dEprR1I3Z3NhRWR4dDFBZkRMNzdQdE9IRHJ6MnRXRmZhenR0UVFveWhlZ0NycnNvcVVxSVFZdjlkRHNTcGRUZXdVTDMweW50clR3R0tWNkNLS2hQWGx0SlVSTFVpb09BaUxVTm1ENk8lMkIlMkYzbkclMkZyRHZwTlF6SEQ0TjB6VyUyRkpzejhNWTNxbTJYUUtRZHpHQlJZRWFKZDlXMUtDNTRzMmJ3QVpRY2Z6djNEd25mZm5VWmw4RERHRHFxYTlZRTRTRnRFS0RIZDBEVGJzSHclM0QlM0Q;OptanonConsent=isGpcEnabled=0&datestamp=Wed+Jan+15+2025+14%3A09%3A47+GMT%2B0000+(Greenwich+Mean+Time)&version=202401.2.0&browserGpcFlag=0&isIABGlobal=false&hosts=&consentId=d5c671ec-7a60-4870-9ab6-51987e9364bc&interactionCount=1&landingPath=NotLandingPage&groups=C0002%3A1%2CC0003%3A1%2CC0004%3A1%2CC0001%3A1&geolocation=GB%3BNIR&AwaitingReconsent=false;new_session=1736950187705; _uetsid=3364a450d27811ef9a5447e7c1aee59b;_uetvid=1e461190b22311efa551c388b7f2c2bd; _ga_ES608WDWW0=GS1.1.1736933221.4.1.1736950197.49.0.0; _ga_GC68N2VLWC=GS1.1.1736950186.8.1.1736950197.49.0.0; _ga_RB305BZ9PQ=GS1.1.1736950186.3.1.1736950197.0.0.1070932008
priority: u=1, i
referer:https://www.cef.co.uk/catalogue/products/5313783-m18stso-ob-18v-force-logic-strut-shears-body-only
sec-ch-ua: "Google Chrome";v="131", "Chromium";v="131", "Not_ABrand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
x-csrf-token:ZUKMwE0PpY+sEk1a44bIkiIQttb1Y++tUnMRPDhsesFiBU0kT3LuvtjlWBV3eNcbFvZPEZ0UM6rrEp8G/ZO4Fw==

Postman curl #

curl --location 'https://www.cef.co.uk/basket_line/validation?id=5313783&quantity=1' \
--header 'Authorization: bearer ZWUyMWZkZmEtMTUxYi00MjE1LWI5N2' \
--header 'Cookie: __cf_bm=9Cgcl1eH0UXsxbGPfA0XyMsDoHUHemkPKxIAUJaeh5o-1736933559-1.0.1.1-.u3iaiQR2owrm7LJCdO4U6RUTasTdjUwuyvPOIm9dDsYs4FPzsGzGPNRj9M7TeUwyCRELkPoLmeD5K0HQOCCjA; _session_id=8fdd44c7c952748339e87d8ada072212'

Postman响应 #

{
"errors": {},
"product_name": "M18STSO-OB 18V FORCE LOGIC™ Strut Shears (Body Only)",
"step_id": "5313783",
"stock_code": "2807-6716",
"part_code": "4933492097",
"max_order_quantity": 99999,
"min_order_quantity": "1.0",
"thumbnail": "https://assets.cef.co.uk/images/pdg/milwaukee_4933492097-a/64x64/milwaukee_4933492097-a.jpg",
"stock_level": 4,
"total_basket_and_requested_quantity": 1.0,
"product_prices": {
    "5313783": {
        "per_price": "2249.95",
        "per_pack": 1,
        "quantity": 1.0,
        "price": 2249.95,
        "price_inclusive": 2699.94,
        "unit": "Each",
        "breaks": {
            "data": []
        },
        "per_price_inc_vat": "2699.94"
    }
}
}

我的Python代码 #

import requests
import pandas as pd
from bs4 import BeautifulSoup

def run(response):
    print(response)
    #soup = BeautifulSoup(response.content, "html.parser")

cookies = {
    '__cf_bm': '2vzCxFIr3PMeDFdYbdwLu52eJvTL_qI4CcngB6h1Yqo-1736931568-1.0.1.1-ENSmZuqsZsb2PjXx._XK1Ce3.rP9Y7qUOLY.1uEDWuWLJFGJfRhM08NR7.Ot5hChoFSwH5q8H_5Vo.W0.WnA_g',
    '_session_id': '8fdd44c7c952748339e87d8ada072212',
}
headers = {
    'accept':'application/json, text/plain, */*',
    'accept-encoding':'gzip, deflate, br, zstd',
    'accept-language':'en-GB,en-US;q=0.9,en;q=0.8',
    'Authorization': 'bearer ZWUyMWZkZmEtMTUxYi00MjE1LWI5N2',
    'priority':'u=1, i',
    'referer':'https://www.cef.co.uk/catalogue/products/5313783-m18stso-ob-18v-force-logic-strut-shears-body-only',
    'sec-ch-ua':'"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"',
    'sec-ch-ua-mobile':'?0',
    'sec-ch-ua-platform':'"Windows"',
    'sec-fetch-dest':'empty',
    'sec-fetch-mode':'cors',
    'sec-fetch-site':'same-origin',
    'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
    'x-csrf-token':'Y4mXSaiglU8PC8AfwsLZRrMcEU5nhXasyovT2LKy+Y9kzlatqt3efnv81VBWPMbPh/roiQ/yqqtz6l3id007WQ=='
    # 'Cookie': '__cf_bm=2vzCxFIr3PMeDFdYbdwLu52eJvTL_qI4CcngB6h1Yqo-1736931568-1.0.1.1-ENSmZuqsZsb2PjXx._XK1Ce3.rP9Y7qUOLY.1uEDWuWLJFGJfRhM08NR7.Ot5hChoFSwH5q8H_5Vo.W0.WnA_g; _session_id=8fdd44c7c952748339e87d8ada072212',
}
params = {
    'id': '5313783',
    'quantity': '1',
}
response = requests.get('https://www.cef.co.uk/basket_line/validation', params=params, cookies=cookies, headers=headers)
run(response)

代码运行后输出 <Response [403]>，求解决思路。

• 校验Cookie有效性：__cf_bm是Cloudflare的临时Cookie，有效期通常只有几分钟，你代码里的__cf_bm和Postman中的不一致且可能已过期，直接复制Postman当前有效的__cf_bm替换代码中的对应值后再测试。
• 检查Authorization令牌完整性：Postman中的Authorization令牌看起来被截断（正常JWT令牌更长），确认是否复制完整，缺失部分会导致权限验证失败。
• 统一Cookie传递方式：不要同时在cookies参数和headers中传递Cookie，建议仅使用cookies字典传递，避免格式冲突。
• 匹配请求头细节：对比浏览器和代码的请求头，确保sec-ch-ua中的Not_ABrand和代码里的Not_A Brand一致（注意空格），这类细微差异可能被反爬机制识别。
• 模拟真实访问流程：先请求商品详情页（即referer对应的URL），获取最新的_session_id、x-csrf-token和__cf_bm后，再发起价格验证请求，模拟浏览器的正常访问顺序。
• 获取详细错误信息：添加allow_redirects=False参数，避免自动重定向干扰，同时打印response.text或response.headers，查看服务器返回的具体错误内容，帮助定位问题。

内容的提问来源于stack exchange，提问作者Matthew Wallace