如何提升Scrapy-Playwright的爬取性能？

测试Scrapy标准请求与Scrapy-Playwright请求爬取http://ipinfo.io/ip，循环次数设为10000次，parse_ipinfo仅记录response.body。结果性能差距极大：

• Playwright请求初始仅25页/分钟
• 标准Scrapy请求达2200页/分钟

因业务需求必须使用Playwright，将REACTOR_THREADPOOL_MAXSIZE从默认10调至100后，Playwright请求性能提升至约90页/分钟，但仍需进一步优化，寻求可行方案。

Playwright 请求代码 #

def start_requests(self):
    for i in range(0, self.count):
        yield scrapy.Request(
            "http://ipinfo.io/ip", 
            callback=self.parse_ipinfo,
            dont_filter=True,
            meta={
                "playwright": True,
            },
        )

标准 Scrapy 请求代码 #

def start_requests(self):
    for i in range(0, self.count):
        yield scrapy.Request(
            "http://ipinfo.io/ip", 
            callback=self.parse_ipinfo,
            dont_filter=True,
        )

• 复用浏览器上下文与页面：默认每个Playwright请求会创建全新页面，开销极大。可在Spider初始化时创建全局浏览器上下文，复用该上下文生成页面，甚至维护页面池循环处理请求，避免重复初始化资源。示例：

  from scrapy_playwright.page import PageCoroutine
  
  def __init__(self, *args, **kwargs):
      super().__init__(*args, **kwargs)
      self.playwright_context = None
  
  def start_requests(self):
      # 先创建上下文（仅执行一次）
      yield scrapy.Request(
          "about:blank",
          callback=self.init_context,
          meta={"playwright": True},
          dont_filter=True,
      )
  
  def init_context(self, response):
      self.playwright_context = response.meta["playwright_context"]
      # 拦截非必要资源加载
      self.playwright_context.route("**/*", self.intercept_request)
      # 生成实际请求，复用上下文
      for i in range(0, self.count):
          yield scrapy.Request(
              "http://ipinfo.io/ip",
              callback=self.parse_ipinfo,
              dont_filter=True,
              meta={
                  "playwright": True,
                  "playwright_context": self.playwright_context,
                  "playwright_page_goto_kwargs": {"wait_until": "commit"},
              },
          )
  
  async def intercept_request(self, route, request):
      # 阻止图片、样式、脚本等非必要资源加载
      if request.resource_type in ["image", "stylesheet", "script", "font"]:
          await route.abort()
      else:
          await route.continue_()
  

• 调整Playwright并发配置：在Scrapy配置文件中修改以下参数，提升并发能力：

  # 每个上下文允许的最大页面数
  PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 50
  # 并发上下文数量
  PLAYWRIGHT_CONCURRENT_CONTEXT = 2
  # 调整Scrapy全局并发数
  CONCURRENT_REQUESTS = 100
  CONCURRENT_REQUESTS_PER_DOMAIN = 100
  

• 优化浏览器启动参数：添加性能优化参数，减少浏览器资源占用：

  # 在settings.py中配置
  PLAYWRIGHT_BROWSER_TYPE = "chromium"
  PLAYWRIGHT_LAUNCH_OPTIONS = {
      "headless": True,
      "args": [
          "--disable-gpu",
          "--disable-dev-shm-usage",
          "--no-sandbox",
          "--disable-setuid-sandbox",
          "--single-process",
          "--disable-accelerated-2d-canvas",
          "--disable-features=VizDisplayCompositor",
      ],
  }
  

• 最小化页面等待时间：针对ipinfo.io/ip这类纯文本响应，设置最严格的页面加载等待策略，避免不必要的等待：

  # 在请求meta中添加
  "playwright_page_goto_kwargs": {"wait_until": "commit"}
  

  commit表示浏览器收到响应头后即停止等待，比默认的load或domcontentloaded更快。

• 禁用Playwright自动页面关闭：默认请求完成后会关闭页面，可设置meta={"playwright": True, "playwright_keep_page": True}，手动管理页面生命周期，复用页面处理多个请求。

内容的提问来源于stack exchange，提问作者samuel guedon