在Lambda上运行Playwright+Scrapy时遇Reactor不兼容问题求助

1. 核心问题修复：提前切换Twisted反应器 #

错误根源是Lambda环境默认初始化了EPollReactor，但Playwright依赖AsyncioSelectorReactor。必须在任何Scrapy模块导入前强制安装Asyncio反应器，否则Twisted会锁定默认反应器导致切换失败。

在Lambda处理函数最开头添加以下代码：

from twisted.internet import asyncioreactor
asyncioreactor.install()

2. 配置Scrapy适配Playwright #

在项目的settings.py中完善Playwright相关配置，适配Lambda无桌面环境：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "args": [
        "--no-sandbox",
        "--disable-dev-shm-usage",
        "--disable-gpu",
        "--single-process",
    ],
}

3. 调整Lambda爬虫启动逻辑 #

确保所有爬虫（含非Playwright爬虫）在统一反应器环境下运行，示例处理函数结构：

# 第一步：先安装Asyncio反应器
from twisted.internet import asyncioreactor
asyncioreactor.install()

# 再导入Scrapy相关模块
from scrapy.crawler import CrawlerProcess
from scrapy.utils.log import configure_logging
from myspiders.embla import EmblaScraper
from myspiders.normal_spider import NormalSpider

def lambda_handler(event, context):
    configure_logging()
    process = CrawlerProcess(settings={
        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        # 其他全局Scrapy设置...
    })

    # 添加所有需运行的爬虫
    process.crawl(EmblaScraper)
    process.crawl(NormalSpider)

    # 启动爬虫直至全部完成
    process.start()
    return {"status": "success", "message": "所有爬虫执行完毕"}

关键注意事项 #

• 反应器初始化顺序不可颠倒，必须在Scrapy模块导入前执行，否则会触发已初始化反应器无法切换的报错。
• Playwright在Lambda上部署时，需确保打包的依赖包含Linux环境对应的Playwright二进制文件（可通过Lambda层或打包时自动安装）。
• 非Playwright爬虫无需修改代码，AsyncioSelectorReactor完全兼容普通Scrapy爬虫的运行逻辑。

内容的提问来源于stack exchange，提问作者KnowHoper