Scrapy-Playwright爬取仅成功3个URL，48个遇TimeoutError求助

问题核心判断 #

不用Playwright时请求正常，说明目标站点对Playwright驱动的浏览器存在反爬限制，少数请求成功大概率是初始请求未触发拦截规则，后续批量请求触发了频率或浏览器指纹检测。

具体解决方案 #

• 调整页面加载策略与超时配置
  默认的load等待策略会等所有资源加载完成，容易超时。改用domcontentloaded或networkidle，同时缩短超时时间：

  # 单个请求配置
  yield scrapy.Request(
      product_url,
      meta={
          'playwright': True,
          'playwright_page_load_strategy': 'domcontentloaded',
          'playwright_navigation_timeout': 30000  # 30秒超时
      },
      callback=self.parse_product
  )
  # 全局配置（settings.py）
  PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30000
  

• 伪装浏览器指纹，规避自动化检测
  Playwright默认的浏览器特征容易被识别，通过上下文配置和JS注入伪装成普通用户：

  def process_request(self, request, spider):
      if request.meta.get('playwright'):
          # 配置上下文参数
          request.meta['playwright_context_kwargs'] = {
              'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
              'viewport': {'width': 1920, 'height': 1080},
              'extra_http_headers': {
                  'Accept-Language': 'en-US,en;q=0.9',
                  'Referer': 'https://wearpact.com/men/apparel'
              }
          }
          # 注入JS移除自动化标识
          request.meta['playwright_on_page_load'] = lambda page: page.add_init_script("""
              Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
              Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']});
          """)
  

• 严格控制并发请求频率
  站点可能限制了IP请求频率，降低并发数并添加延迟：

  # settings.py
  PLAYWRIGHT_MAX_CONTEXTS = 2  # 同时运行的浏览器实例数
  PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 3  # 每个实例的最大页面数
  DOWNLOAD_DELAY = 2  # 请求间隔2秒
  CONCURRENT_REQUESTS = 4  # 全局并发数
  

• 自动关闭邮件捕获弹窗
  弹窗会阻塞页面流程，添加自动关闭逻辑：

  request.meta['playwright_on_page_load'] = lambda page: asyncio.gather(
      page.add_init_script("""
          Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
      """),
      # 尝试关闭弹窗，没找到就忽略
      page.wait_for_selector('button[aria-label="Close"]', timeout=5000).then(lambda btn: btn.click()).catch(lambda _: None)
  )
  

• 拦截非必要资源，加速页面加载
  过滤图片、视频等资源，减少加载时间：

  request.meta['playwright_on_context'] = lambda context: context.route(
      '**/*.{png,jpg,jpeg,gif,webp,mp4}',
      lambda route: route.abort()
  )
  

• 开启详细日志排查超时细节
  开启Playwright DEBUG日志，定位具体超时环节：

  # settings.py
  PLAYWRIGHT_LOG_LEVEL = 'DEBUG'
  

内容的提问来源于stack exchange，提问作者marcb