Python Requests调用API报403错误，浏览器正常如何优化？

你的问题核心是API的反爬机制识别出了Python请求并非真实浏览器发起，以下是几个可实现自动化的优化方案：

1. 用Requests Session自动维护会话Cookie #

手动复制Cookie的弊端在于Cookie会过期，而requests.Session()可以模拟浏览器的会话流程，自动获取并更新必要的Cookie。先访问Rewe移动端首页触发Cookie生成，再请求目标API：

import requests
import time
import random

session = requests.Session()

# 先访问首页获取初始会话Cookie
session.get("https://mobile.rewe.de/")
time.sleep(random.uniform(1, 2))  # 模拟真实页面加载延迟

def getoffers():
    urlstring = "https://mobile-api.rewe.de/api/v3/all-offers?marketCode=3200008"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
        "Accept": "application/json, text/plain, */*",  # 匹配API实际接受的内容类型，可从浏览器请求头中复制
        "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8,de-DE;q=0.7,de;q=0.6",
        "Referer": "https://mobile.rewe.de/",
        "Origin": "https://mobile.rewe.de"
    }
    response = session.get(urlstring, headers=headers)
    response.raise_for_status()  # 抛出请求错误方便调试
    return response.json()["categories"][0]["offers"]

2. 绕过Cloudflare验证（针对cf_clearance Cookie） #

你Cookie中的cf_clearance是Cloudflare人机验证后的凭证，普通Requests请求无法直接生成。可以用undetected-chromedriver模拟真实浏览器绕过验证，获取有效Cookie后再发起请求：

from undetected_chromedriver import Chrome, ChromeOptions
import requests
import time
import random

def get_cloudflare_cookies():
    options = ChromeOptions()
    options.add_argument("--headless=new")  # 无头模式，不显示浏览器窗口
    options.add_argument("--disable-gpu")
    with Chrome(options=options) as driver:
        driver.get("https://mobile-api.rewe.de/api/v3/all-offers?marketCode=3200008")
        time.sleep(random.uniform(2, 4))  # 等待验证完成
        # 转换Cookie格式为Requests可用的字典
        cookie_dict = {cookie['name']: cookie['value'] for cookie in driver.get_cookies()}
        return cookie_dict

def getoffers():
    urlstring = "https://mobile-api.rewe.de/api/v3/all-offers?marketCode=3200008"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
        "Accept": "application/json, text/plain, */*",
        "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8,de-DE;q=0.7,de;q=0.6",
        "Referer": "https://mobile.rewe.de/",
        "Origin": "https://mobile.rewe.de"
    }
    cookies = get_cloudflare_cookies()
    response = requests.get(urlstring, headers=headers, cookies=cookies)
    response.raise_for_status()
    return response.json()["categories"][0]["offers"]

3. 完善请求头细节 #

对比浏览器的真实请求头，补充缺失的现代浏览器专属字段，进一步降低被识别的概率：

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8,de-DE;q=0.7,de;q=0.6",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://mobile.rewe.de/",
    "Origin": "https://mobile.rewe.de",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-site",
    "Connection": "keep-alive"
}

4. 控制请求频率 #

频繁请求会触发反爬机制，添加随机延迟模拟真实用户操作间隔：

# 在每次API请求前添加随机延迟
time.sleep(random.uniform(1, 3))

注意事项 #

• 严格遵守Rewe网站的使用条款与robots.txt规则，避免过度请求导致IP被封禁。
• 如果API需要用户认证，需额外模拟登录流程获取会话凭证。

内容的提问来源于stack exchange，提问作者Husky-CGN