Scrapy Playwright Chromium无头模式报403，非无头正常求解决方案

• 使用Scrapy-Playwright抓取Barnes & Noble电商站点时，Chromium启用headless:True会返回403错误，headless:False时返回正常200状态码
• 已尝试随机化User-Agent，但仍被反爬拦截
• 当前改用Firefox和WebKit驱动，但速度过慢，希望换回Chromium
• 猜测可能是Chromium版本导致的问题，但不清楚如何在Playwright中降级Chromium

注：ISBN为图书编码

相关代码 #

def make_request_from_data(self, data):
    payload = json.loads(data)
    isbn = payload["isbn"]

    url = f"https://www.barnesandnoble.com/s/{isbn}"
    meta = {
        "region": self.region,
        "isbn": isbn,
        "playwright": True,
        "playwright_include_page": True,
        "playwright_context": f"context-{isbn}",
        "playwright_context_kwargs": {
            "java_script_enabled": True,
        },
    }
    headers = {
        "accept-encoding": "gzip, deflate, br",
        "accept-language": "en",
        "cache-control": "no-cache",
        "pragma": "no-cache",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "none",
        "sec-fetch-user": "?1",
        "upgrade-insecure-requests": "1",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    }  
    yield Request(
        headers=headers,
        url=url, 
        callback=self.parse,
        errback=self.close_context_on_error,
        meta=meta,
        dont_filter=True,
    )

一、绕过无头模式反爬特征（优先推荐，无需降级） #

Chromium旧无头模式存在明显反爬识别特征，优先尝试以下优化：

1. 启用Playwright新无头模式
   在playwright_context_kwargs中设置headless="new"，该模式更接近真实有头浏览器，降低被识别概率：

"playwright_context_kwargs": {
    "java_script_enabled": True,
    "headless": "new",  # 使用新无头模式
    "viewport": {"width": 1920, "height": 1080},  # 模拟桌面端视口
    "locale": "en-US",
}

2. 注入脚本隐藏navigator.webdriver标识
   在解析函数中，页面初始化前注入脚本覆盖webdriver属性：

async def parse(self, response):
    page = response.meta["playwright_page"]
    # 注入脚本绕过webdriver检测
    await page.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
        // 可选：补充插件数量模拟真实浏览器
        Object.defineProperty(navigator, 'plugins', {
            get: () => [1, 2, 3],
        });
    """)
    # 等待页面稳定加载
    await page.wait_for_load_state("networkidle")
    # 后续解析逻辑...

3. 统一上下文与请求头
   将请求头传入Playwright上下文的extra_http_headers，确保上下文与Request的请求头一致：

"playwright_context_kwargs": {
    "java_script_enabled": True,
    "headless": "new",
    "viewport": {"width": 1920, "height": 1080},
    "extra_http_headers": headers,  # 复用请求头
}

二、降级Playwright中的Chromium（若上述方法无效） #

如果确认是Chromium版本问题，可按以下步骤操作：

1. 查看可用Chromium版本
   执行命令查看Playwright支持的旧版本列表：

playwright browsers list

2. 安装指定版本的Chromium
   选择一个较旧的稳定版本，执行安装命令（示例版本为118.0.5993.70）：

playwright install chromium=118.0.5993.70

3. 在Scrapy中指定Chromium路径
   在项目settings.py中配置已安装的旧版本Chromium路径：

PLAYWRIGHT_BROWSERS = {
    "chromium": {
        "executable_path": "~/.cache/ms-playwright/chromium-1180/chrome-linux/chrome",  # 替换为实际路径
    }
}

注：路径可根据系统调整，一般在~/.cache/ms-playwright/（Linux/macOS）或%USERPROFILE%\.cache\ms-playwright\（Windows）目录下。

内容的提问来源于stack exchange，提问作者aditya narayanan