Scrapy集成Playwright时出现KeyError: 'playwright_page'的原因及解决方法

在使用Scrapy结合Playwright开发爬虫时，编写了如下代码：

async def errback_close_page(self, failure):
        page = failure.request.meta["playwright_page"]
        await page.close()

def start_requests(self):
        if not self.start_urls and hasattr(self, "start_url"):
            raise AttributeError(
                "Crawling could not start: 'start_urls' not found "
                "or empty (but found 'start_url' attribute instead, "
                "did you miss an 's'?)"
            )
        for url in self.start_urls:
            npo = self.npos[url]
            logging.info("### crawl: %s", url)
            yield scrapy.Request(
                url, callback=self.my_parse, dont_filter=True,meta={"playwright": True, "playwright_include_page": True, 'start_time': datetime.utcnow()}, cb_kwargs={"npo": npo},errback= self.errback_close_page
                
            )

解析函数代码：

async def my_parse(self, response, npo):
    page = response.meta["playwright_page"]

运行时触发如下报错：

Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1065, in adapt
    extracted = result.result()
  File "/home/ec2-user/SageMaker/xx", line 50, in errback_close_page
    page = failure.request.meta["playwright_page"]
KeyError: 'playwright_page'

请问出现该错误的原因是什么，该如何修复？

• playwright_page 是Scrapy-Playwright中间件成功初始化页面后，才会添加到请求meta中的字段。如果请求在Playwright中间件处理前就失败（比如DNS解析失败、网络连接超时、请求被Scrapy下载器直接拦截），此时meta里还没有这个字段，直接通过键取值就会触发KeyError。
• 即使Playwright中间件开始处理请求，页面初始化过程中也可能出现异常，导致playwright_page还未被写入meta，同样会引发报错。

修改错误回调函数，先检查meta中是否存在playwright_page，存在时再执行关闭操作：

async def errback_close_page(self, failure):
    page = failure.request.meta.get("playwright_page")
    if page is not None:
        await page.close()

也可以用异常捕获的方式处理：

async def errback_close_page(self, failure):
    try:
        page = failure.request.meta["playwright_page"]
        await page.close()
    except KeyError:
        # 页面未初始化，无需处理
        pass

另外，要确保Scrapy-Playwright中间件已正确配置在settings.py中：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

SPIDER_MIDDLEWARES = {
    "scrapy_playwright.middleware.PlaywrightMiddleware": 543,
}

内容的提问来源于stack exchange，提问作者Nekender Shekhawat