Scrapy爬取Mercadona网站无数据，设ROBOTSTXT_OBEY=False仍无效求解决

你的猜测没错，Mercadona这类电商网站通常有反爬机制，结合你的情况，按以下步骤排查解决：

• 添加真实请求头
  默认Scrapy的请求头容易被识别，先在settings.py里配置真实的User-Agent和其他必要头：

  DEFAULT_REQUEST_HEADERS = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'Accept-Language': 'es-ES,es;q=0.9',
  }
  

• 验证Cookie会话
  部分网站会要求先建立会话（比如访问首页获取Cookie）再请求搜索页，修改爬虫用start_requests实现：

  def start_requests(self):
      # 先请求首页获取Cookie
      yield scrapy.Request(
          url='https://tienda.mercadona.es/',
          callback=self.fetch_search_page,
          dont_filter=True
      )
  
  def fetch_search_page(self, response):
      # 携带首页获取的Cookie请求搜索页
      yield scrapy.Request(
          url='https://tienda.mercadona.es/search-results?query=leche%20entera',
          callback=self.parse
      )
  
  def parse(self, response):
      # 先打印响应内容确认是否拿到有效页面
      print(response.text[:500])
      sample = response.css("div").get()
      yield {'name': sample}
  

• 处理动态渲染内容
  如果网站用JS加载商品数据，Scrapy默认无法获取动态内容，需要用Playwright渲染页面：

  1. 安装依赖：pip install scrapy-playwright
  2. 在settings.py启用中间件：

     DOWNLOAD_HANDLERS = {
         "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
         "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
     }
     SPIDER_MIDDLEWARES = {
         "scrapy_playwright.middleware.PlaywrightMiddleware": 543,
     }
     PLAYWRIGHT_LAUNCH_OPTIONS = {
         "headless": False,  # 测试时可关闭无头模式查看页面状态
     }
     

  3. 修改爬虫使用Playwright请求：

     def start_requests(self):
         yield scrapy.Request(
             url='https://tienda.mercadona.es/search-results?query=leche%20entera',
             meta={"playwright": True},
             callback=self.parse
         )
     

• 先调试响应内容
  不管用哪种方法，先在parse方法里打印response.text的前几百字符，确认是拿到了目标页面还是反爬拦截页面（比如验证码、访问限制提示），再针对性调整。

内容的提问来源于stack exchange，提问作者M. Mariscal