使用Scrapy爬取ResearchGate返回‘Just a moment’无法获取作者页面

针对你遇到的反爬拦截问题，以下是实际可落地的解决步骤：

1. 替换Splash为Playwright处理动态渲染与Cookie验证 #

ResearchGate的反爬机制会严格校验浏览器环境，Splash的模拟度不足以绕过验证，Playwright能完全模拟真实浏览器的行为，包括JS渲染、自动维护会话Cookie：

• 安装依赖：

  pip install scrapy-playwright
  playwright install chromium
  

• 在Scrapy项目的settings.py中配置：

  DOWNLOAD_HANDLERS = {
      "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
      "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
  }
  PLAYWRIGHT_LAUNCH_OPTIONS = {
      "headless": True,  # 调试时可改为False查看浏览器实际加载过程
      "args": ["--no-sandbox", "--disable-setuid-sandbox"],
  }
  PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 60000  # 延长超时时间，确保页面完全加载
  

• 编写爬虫示例：

  import scrapy
  from scrapy_playwright.page import PageCoroutine
  
  class ResearchgateSpider(scrapy.Spider):
      name = "researchgate"
      start_urls = ["https://www.researchgate.net/scientific-contributions/Gregory-Phelan-2126234043"]
  
      def start_requests(self):
          for url in self.start_urls:
              yield scrapy.Request(
                  url,
                  meta={
                      "playwright": True,
                      "playwright_page_coroutines": [
                          PageCoroutine("wait_for_selector", "div.co-author-list"),  # 等待合著者列表元素加载完成
                      ],
                  },
              )
  
      def parse(self, response):
          # 提取当前页面的合著者链接
          co_author_links = response.css("div.co-author-list a::attr(href)").getall()
          for link in co_author_links:
              full_url = response.urljoin(link)
              yield scrapy.Request(
                  full_url,
                  meta={
                      "playwright": True,
                      "playwright_page_coroutines": [
                          PageCoroutine("wait_for_selector", "div.co-author-list"),
                      ],
                  },
                  callback=self.parse,
              )
          # 提取当前作者的基础信息（按需扩展）
          author_name = response.css("h1::text").get().strip()
          yield {"author_name": author_name, "profile_url": response.url}
  

2. 手动注入有效Cookie（可选，提升通过率） #

如果仍遇到验证拦截，可手动从浏览器获取有效Cookie注入：

• 用Chrome打开目标URL，等待页面完全加载后，按F12打开开发者工具→Application→Cookies，复制所有Cookie的键值对。
• 在爬虫的start_requests方法中添加Cookie配置：

  def start_requests(self):
      # 替换为你从浏览器复制的Cookie
      custom_cookies = {
          "RG_SESSION_ID": "你的会话ID",
          "rg_mid": "对应Cookie值",
          # 其他Cookie键值对
      }
      for url in self.start_urls:
          yield scrapy.Request(
              url,
              cookies=custom_cookies,
              meta={"playwright": True, ...},
          )
  

3. 严格控制爬取频率，模拟人类行为 #

• 在settings.py中配置爬取间隔：

  DOWNLOAD_DELAY = 3
  RANDOMIZE_DOWNLOAD_DELAY = True
  CONCURRENT_REQUESTS = 1  # 单线程爬取，降低触发反爬的概率
  

• 避免一次性批量爬取大量页面，可分批次执行爬虫任务。

4. 确认爬取合规性 #

爬取前务必查看ResearchGate的robots.txt和官方使用条款，确保你的爬取行为符合平台规定，避免法律风险。

内容的提问来源于stack exchange，提问作者Stu_gro