Scrapy-playwright报错KeyError: 'playwright_page'问题求助

检查核心配置是否生效 #

• 确认settings.py中已启用Playwright中间件：

  DOWNLOADER_MIDDLEWARES = {
      'scrapy_playwright.middleware.PlaywrightMiddleware': 543,
  }
  

• 确保已开启Playwright下载器：

  DOWNLOAD_HANDLERS = {
      "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
      "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
  }
  

验证请求Meta参数的正确性 #

• 发起请求时，必须在meta中同时指定playwright和playwright_include_page，缺少前者会导致Playwright中间件不处理该请求：

  yield scrapy.Request(
      url=your_target_url,
      callback=self.parse,
      meta={
          'playwright': True,
          'playwright_include_page': True,
          # 可选：添加滚动加载逻辑
          'playwright_page_goto_kwargs': {'wait_until': 'networkidle'},
          'playwright_page_evaluate': """
              await new Promise(resolve => {
                  const scroll = () => {
                      if (document.body.scrollHeight > window.scrollY + window.innerHeight) {
                          window.scrollBy(0, window.innerHeight);
                          setTimeout(scroll, 1000);
                      } else {
                          resolve();
                      }
                  };
                  scroll();
              });
          """
      }
  )
  

排查中间件优先级冲突 #

• 确保Playwright中间件的优先级（543）高于其他自定义下载中间件，避免请求被提前拦截处理，导致playwright_page未注入到response.meta中。

检查版本兼容性 #

• 运行以下命令确认Scrapy与Scrapy-Playwright版本匹配，建议使用最新稳定版：

  pip list | grep -E "scrapy|scrapy-playwright"
  

调试验证 #

• 在parse方法开头添加调试代码，确认response.meta内容：

  def parse(self, response):
      print(response.meta.keys())  # 查看是否包含playwright_page
      page = response.meta.get('playwright_page')
      if page:
          # 处理页面逻辑后关闭页面，避免资源泄漏
          await page.close()
      else:
          print("playwright_page not found in response.meta")
  

内容的提问来源于stack exchange，提问作者CoolMathematician