Scrapy爬取新楼盘页面返回None，请求排查解决

1. 先确认页面内容是否为动态加载 #

• 打开目标页面，右键→查看网页源代码，搜索你要抓取的目标内容。如果搜索不到，说明内容是通过JavaScript动态渲染的，Scrapy默认获取的初始HTML里没有这部分内容。
• 用Scrapy Shell快速验证：

  scrapy shell "https://xn--80az8a.xn--d1aqf.xn--p1ai/%D1%81%D0%B5%D1%80%D0%B2%D0%B8%D1%81%D1%8B/%D0%BA%D0%B0%D1%82%D0%B0%D0%BB%D0%BE%D0%B3-%D0%BD%D0%BE%D0%B2%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BA/%D1%81%D0%BF%D0%B8%D1%81%D0%BE%D0%BA-%D0%BE%D0%B1%D1%8A%D0%B5%D0%BA%D1%82%D0%BE%D0%B2/%D1%81%D0%BF%D0%B8%D1%81%D0%BE%D0%BA?place=0-1&objStatus=0"
  

  执行response.body.decode('utf-8')后搜索目标内容，若仍无结果，即可确认是动态加载问题。

2. 动态加载内容的处理方案 #

方案一：用Scrapy-Playwright渲染JS #

• 安装依赖：

  pip install scrapy-playwright
  

• 在settings.py中配置：

  DOWNLOAD_HANDLERS = {
      "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
      "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
  }
  PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": True}
  

• 修改爬虫代码：

  import scrapy
  from scrapy_playwright.page import PageCoroutine
  
  class NewObjectsSpider(scrapy.Spider):
      name = "new_objects"
      start_urls = ["https://xn--80az8a.xn--d1aqf.xn--p1ai/%D1%81%D0%B5%D1%80%D0%B2%D0%B8%D1%81%D1%8B/%D0%BA%D0%B0%D1%82%D0%B0%D0%BB%D0%BE%D0%B3-%D0%BD%D0%BE%D0%B2%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BA/%D1%81%D0%BF%D0%B8%D1%81%D0%BE%D0%BA-%D0%BE%D0%B1%D1%8A%D0%B5%D0%BA%D1%82%D0%BE%D0%B2/%D1%81%D0%BF%D0%B8%D1%81%D0%BE%D0%BA?place=0-1&objStatus=0"]
  
      def start_requests(self):
          for url in self.start_urls:
              yield scrapy.Request(
                  url,
                  meta={
                      "playwright": True,
                      "playwright_page_coroutines": [PageCoroutine("wait_for_selector", "你的目标CSS选择器")]
                  },
              )
  
      def parse(self, response):
          # 尝试提取内容
          target_items = response.css("你的目标CSS选择器").extract()
          print(target_items)
  

方案二：直接抓取数据接口 #

打开浏览器开发者工具→网络→XHR/Fetch标签，找到加载目标数据的API接口，直接请求该接口获取数据，这种方式比渲染JS更高效。

3. 检查CSS选择器是否正确 #

• 避免类名空格错误：如果元素类是card item，CSS选择器要写成.card.item，而非.card item。
• 检查是否在iframe内：若目标内容嵌套在iframe中，需先切换到iframe再提取内容。
• 用浏览器工具复制正确选择器：右键目标元素→检查→右键元素代码→复制→复制选择器，直接使用该选择器测试。

4. 反爬机制排查 #

• 设置浏览器UA：在settings.py中添加：

  DEFAULT_REQUEST_HEADERS = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
  }
  

• 检查Cookie：如果页面需要登录会话，需在请求中携带对应Cookie。

内容的提问来源于stack exchange，提问作者Клим