Scrapy爬取Tripadvisor遇403及scrapy-user-agents报错求助

• 修复scrapy-user-agents的警告问题
  出现UnsupportedBrowserType或UnsupportedDeviceType警告，是因为该库默认的UA池包含不支持的浏览器/设备类型，直接自定义UA列表或换用更稳定的库即可：

  1. 注释掉settings.py中scrapy-user-agents的中间件配置
  2. 改用scrapy-fake-useragent库（先执行pip install scrapy-fake-useragent），并在settings.py中配置：

  USER_AGENTS = [
      'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
      'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15',
      'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
  ]
  DOWNLOADER_MIDDLEWARES = {
      'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
      'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
  }
  

• 破解Tripadvisor的403反爬限制
  403状态码核心是请求被识别为爬虫，除UA外还要做这些配置：

  1. 补全真实请求头：在爬虫的start_requests方法中添加浏览器完整请求头：

  def start_requests(self):
      headers = {
          'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
          'Accept-Language': 'it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3',
          'Referer': 'https://www.tripadvisor.it/Rome-d187791-Restaurants.html',
          'Upgrade-Insecure-Requests': '1',
          'Accept-Encoding': 'gzip, deflate, br'
      }
      yield scrapy.Request(url=self.start_urls[0], headers=headers, callback=self.parse)
  

  2. 设置请求延迟：在settings.py中配置：

  DOWNLOAD_DELAY = 3
  RANDOMIZE_DOWNLOAD_DELAY = True
  

  3. 启用Cookie管理：确保settings.py中COOKIES_ENABLED = True，让Scrapy自动维护会话Cookie
  4. 配置代理IP：如果IP被封禁，添加代理中间件，使用高匿代理切换IP，示例中间件逻辑：

  # middlewares.py
  class ProxyMiddleware:
      def process_request(self, request, spider):
          request.meta['proxy'] = 'http://your-proxy-ip:port'
  

  并在settings.py中启用：

  DOWNLOADER_MIDDLEWARES.update({
      'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
      'your_project.middlewares.ProxyMiddleware': 100,
  })
  

• 验证请求有效性
  用带UA的scrapy shell测试请求，确认是否能返回200：

  scrapy shell -s USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "https://www.tripadvisor.it/Restaurant_Review-g187791-d25150423-Reviews-Don_s_Meats_Spirits-Rome_Lazio.html"
  

  执行response.status查看状态码，response.text查看页面内容，确认是否正常获取。

内容的提问来源于stack exchange，提问作者Rodolfo