Shopee商品数据API失效求助（原Scrapy爬虫曾可用）

问题背景 #

此前使用Scrapy爬虫调用Shopee菲律宾站api/v4/pdp/get_pc接口获取商品数据，2023年8月初可正常运行，依赖请求头{"x-api-source":"pc","af-ac-enc-dat":"null"}及指定cookies即可成功获取数据。但2023年8月20日起爬虫失效，返回错误：

{"is_customized":false,"is_login":true,"platform":0,"action_type":2,"error":90309999,"tracking_id":"24d95bd5-40e5-44cd-b30b-885711481170","report_extra_info":""}

失效原因 #

Shopee更新了接口验证机制，原请求头缺少必要的浏览器标识字段，且部分cookies已过期或验证逻辑变更，导致请求被反爬系统拦截，返回通用拦截错误码90309999。

修复方案 #

• 更新请求头：添加User-Agent、Referer、Accept-Language等模拟真实PC浏览器的字段，移除无效的af-ac-enc-dat头
• 更新cookies：使用从浏览器访问目标商品页后复制的最新有效cookies，重点确保SPC_SI、SPC_T_ID等核心验证字段未过期
• 优化请求环境：补充必要的HTTP请求字段，降低被识别为爬虫的概率

修改后的Scrapy爬虫代码 #

from scrapy.utils.response import open_in_browser
import scrapy
import json

class TestSpider(scrapy.Spider):
    name = "test-spider"
    allowed_domains = ["shopee.ph"]

    # 替换为你从浏览器获取的最新有效cookies（保持格式一致）
    shopee_cookies = '[{"name": "csrftoken", "value": "YOUR_NEW_CSRF_TOKEN", "domain": "shopee.ph", "path": "/", "expires": -1, "httpOnly": false, "secure": false, "sameSite": "Lax"}, {"name": "SPC_SI", "value": "YOUR_NEW_SPC_SI", "domain": ".shopee.ph", "path": "/", "expires": 1692551246.336331, "httpOnly": true, "secure": true, "sameSite": "Lax"}, {"name": "SPC_T_ID", "value": "YOUR_NEW_SPC_T_ID", "domain": ".shopee.ph", "path": "/", "expires": 1727024846.336355, "httpOnly": true, "secure": true, "sameSite": "Lax"}]'

    shopee_cookies = json.loads(shopee_cookies)

    def start_requests(self):
        target_item_url = "https://shopee.ph/Realme-C53-C55-C35-C21Y-C25Y-RealmeC11-2021-C25-C25S-C15-C12-C2-C17-C3-Realme8-Pro-8i-7i-5-5i-6i-Fashion-Electroplated-Maple-Leaf-Square-Plating-Soft-Silicone-Case-Meiting-i.237078553.6929743700"
        yield scrapy.Request(
            url="https://shopee.ph/api/v4/pdp/get_pc?shop_id=237078553&item_id=6929743700",
            cookies=self.shopee_cookies,
            headers={
                "x-api-source": "pc",
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
                "Referer": target_item_url,
                "Accept-Language": "en-US,en;q=0.9",
                "Accept": "application/json, text/plain, */*"
            },
            callback=self.parse_item,
        )

    def parse_item(self, response):
        open_in_browser(response)
        # 可在此添加数据解析逻辑
        data = json.loads(response.text)
        self.logger.info(f"商品名称：{data.get('data', {}).get('item', {}).get('name')}")

注意事项 #

• cookies定期更新：Shopee核心验证cookies有过期时间，需定期从浏览器复制更新
• 控制请求频率：在settings.py中添加DOWNLOAD_DELAY = 2，避免因请求频繁触发反爬限制
• 动态调整策略：若再次出现拦截，可尝试添加X-Requested-With头，或使用Scrapy中间件模拟更真实的浏览器行为

内容的提问来源于stack exchange，提问作者Ice Bear