使用Playwright抓取大众瑞典租赁页面时无法拦截请求/响应体

• VPN/代理兼容性与有效性问题：站点仅允许瑞典IP访问，现有Playwright/Selenium未配置有效VPN环境，导致页面无法正常渲染，目标API请求根本没触发；带VPN插件的浏览器启动后冻结，多是自动化工具与VPN插件的兼容性冲突，或是VPN节点稳定性差。
• API访问验证缺失：找到的带动态nocache参数的隐藏API，依赖请求上下文验证（如Cookie、Referer、User-Agent等），直接请求缺少这些必要信息，因此返回403。
• Playwright代码逻辑缺陷：当前代码监听所有请求/响应，未做目标API过滤，输出杂乱易错过关键数据；且wait_until='networkidle'可能过早关闭上下文，导致部分请求未完成就终止。

方案1：给Playwright配置系统级VPN（优先推荐） #

先确保本地系统通过稳定的瑞典VPN节点连接，验证普通浏览器能正常访问目标站点，再修改代码过滤目标请求并延长等待时间：

from playwright.sync_api import sync_playwright

def capture_target_response(response):
    # 根据实际API特征调整过滤规则，比如包含"leasing"或"model"的URL
    if "leasing" in response.url and response.status == 200:
        try:
            data = response.json()
            print("目标车型租赁数据：", data)
            # 可在此将数据写入临时文件或直接传入Scrapy Item
        except:
            pass

with sync_playwright() as p:
    # 启动浏览器时继承系统VPN代理
    browser = p.chromium.launch(headless=False, proxy={"server": "auto"})
    # 设置真实浏览器UA，避免被识别为自动化工具
    page = browser.new_page(user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
    
    page.on("response", capture_target_response)
    
    page.goto("https://privatleasing.volkswagen.se/valjmodell", wait_until='load')
    # 等待页面核心元素加载完成，确保API请求触发
    page.wait_for_selector(".model-list-item")
    # 额外延迟，保证所有异步请求完成
    page.wait_for_timeout(3000)
    
    page.context.close()
    browser.close()

方案2：Scrapy集成Playwright（适配现有项目） #

利用scrapy-playwright插件将Playwright的渲染能力集成到Scrapy项目中，同时配置系统级VPN：

1. 安装依赖：pip install scrapy-playwright
2. 在项目settings.py中启用插件并配置：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": False,
    "proxy": {"server": "auto"},  # 继承系统VPN配置
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
}

3. 编写Spider监听目标响应：

import scrapy
from scrapy_playwright.page import PageCoroutine

class VolkswagenLeasingSpider(scrapy.Spider):
    name = "volkswagen_leasing"
    start_urls = ["https://privatleasing.volkswagen.se/valjmodell"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_page_coroutines": [
                        PageCoroutine("wait_for_selector", ".model-list-item"),
                        PageCoroutine("wait_for_timeout", 3000),
                    ],
                },
            )

    def parse(self, response):
        page = response.meta["playwright_page"]
        # 遍历所有请求的响应，筛选目标API数据
        for req in page.context.request.all_requests:
            if req.response and "leasing" in req.url:
                try:
                    data = req.response.json()
                    yield {"model_name": data.get("model"), "price": data.get("leasing_price")}
                except:
                    pass
        page.close()

方案3：补全API请求验证信息 #

在VPN环境下，通过浏览器开发者工具复制目标API的完整请求头（包括Cookie、Referer、X-Requested-With等），直接用Scrapy发送请求：

import scrapy

class VolkswagenAPISpider(scrapy.Spider):
    name = "volkswagen_api"
    # 替换为实际API地址，动态nocache参数可从页面源码或初始请求中提取
    start_urls = ["https://privatleasing.volkswagen.se/api/vehicle-models?nocache=123456789"]

    def start_requests(self):
        headers = {
            "Cookie": "从浏览器复制的有效Cookie",
            "Referer": "https://privatleasing.volkswagen.se/valjmodell",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
            "X-Requested-With": "XMLHttpRequest"
        }
        for url in self.start_urls:
            yield scrapy.Request(url, headers=headers)

    def parse(self, response):
        data = response.json()
        # 提取车型与价格数据
        for model in data.get("models", []):
            yield {
                "model": model.get("name"),
                "monthly_price": model.get("leasing_price")
            }

内容的提问来源于stack exchange，提问作者Dron22