如何让Scrapy-playwright在出现错误时执行重试？

你的问题出在Scrapy-Playwright的下载处理器默认未将部分异常传递给Scrapy的重试中间件，且默认重试规则未覆盖Playwright抛出的异常类型。按以下步骤调整配置即可解决：

1. 扩展重试异常类型 #

在settings.py中添加RETRY_EXCEPTIONS配置，将Playwright可能抛出的异常纳入重试范围：

RETRY_EXCEPTIONS = (
    # Scrapy原生重试异常
    'scrapy.exceptions.ConnectionError',
    'scrapy.exceptions.TimeoutError',
    # Playwright相关异常
    'playwright._impl._api_types.Error',
    'playwright._impl._api_types.TimeoutError',
    'playwright._impl._api_types.ConnectionError',
    # 系统级网络异常
    ConnectionRefusedError,
    TimeoutError,
)

注：使用字符串形式的异常类路径可避免提前导入的问题，也可直接导入对应类后传入。

2. 确保重试中间件启用 #

确认Scrapy的RetryMiddleware处于启用状态（默认已启用，若手动关闭过需重新开启）：

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
}

3. 适配Playwright的超时设置 #

Scrapy原生的DOWNLOAD_TIMEOUT对Playwright请求不生效，需在请求的meta中单独配置Playwright的超时参数，避免因页面加载超时直接终止而不触发重试：
在爬虫请求中添加：

yield scrapy.Request(
    url=your_target_url,
    meta={
        'playwright': {
            'timeout': 60000,  # 单位毫秒，对应60秒
        }
    },
    callback=self.parse
)

4. 修改后的完整settings.py示例 #

RETRY_ENABLED = True
RETRY_TIMES = 3
# 可选：扩展需要重试的HTTP状态码，比如429、403等
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429, 403]

RETRY_EXCEPTIONS = (
    'scrapy.exceptions.ConnectionError',
    'scrapy.exceptions.TimeoutError',
    'playwright._impl._api_types.Error',
    'playwright._impl._api_types.TimeoutError',
    'playwright._impl._api_types.ConnectionError',
    ConnectionRefusedError,
    TimeoutError,
)

DOWNLOAD_TIMEOUT = 60
DOWNLOAD_DELAY = random.uniform(0, 1)

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# 确保重试中间件优先级正确（默认已启用，若有自定义中间件需调整）
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
}

额外说明 #

• 若使用Scrapy-Playwright版本≥0.0.20，部分异常传递逻辑已优化，但仍需显式配置RETRY_EXCEPTIONS覆盖Playwright相关异常。
• 检查爬虫的errback方法是否手动终止了请求，确保错误被正常抛给重试中间件。

内容的提问来源于stack exchange，提问作者Repzz