ScrapyRequest结合Rule与LinkExtractor爬取仅完成第一页后停止求助

1. CrawlSpider规则失效：你重写了parse方法，而CrawlSpider的Rule逻辑依赖默认parse方法来提取链接、生成后续请求，导致规则无法触发，自然不会跟进其他页面。
2. 普通Request无法渲染JS：即使规则生效，默认生成的是普通Scrapy Request，不会经过Selenium渲染，JS动态加载的链接无法被提取，爬虫也无法继续爬取。

方案一：修复CrawlSpider规则，用Selenium处理所有请求 #

自定义process_request函数，让Rule生成SeleniumRequest，同时避免覆盖默认parse方法：

import scrapy
from scrapy_selenium import SeleniumRequest
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

def selenium_process_request(request, response):
    # 将普通Request转为SeleniumRequest，确保所有请求都经过JS渲染
    return SeleniumRequest(
        url=request.url,
        dont_filter=True,
        callback=response.request.callback
    )

class MySpider(CrawlSpider):
    name = 'example'
    start_urls = ['https://www.example.com/']
    rules = (
        Rule(
            LinkExtractor(allow_domains=['example.com']),
            follow=True,
            process_request=selenium_process_request,
            callback='parse_item'  # 自定义页面解析方法，不覆盖默认parse
        ),
    )

    def start_requests(self):
        for url in self.start_urls:
            print("+++++++++++++++++++++++++++++++++++++++++++++++++++++",url)
            yield SeleniumRequest(url=url, callback=self.parse,dont_filter=True)

    def parse_item(self, response):
        print(response.url)
        item = {'url': response.url, 'html': response.body}
        yield item

    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy_selenium.SeleniumMiddleware': 800
        },
        'SELENIUM_DRIVER_NAME': 'chrome',
        'SELENIUM_DRIVER_EXECUTABLE_PATH': '/home/ubuntu/selenium_drivers/chromedriver',
        'SELENIUM_DRIVER_ARGUMENTS': ['-headless']
    }

方案二：改用普通Spider，手动提取链接 #

放弃CrawlSpider的规则，自己维护已访问URL集合，手动生成SeleniumRequest：

import scrapy
from scrapy_selenium import SeleniumRequest
from scrapy.linkextractors import LinkExtractor

class MySpider(scrapy.Spider):
    name = 'example'
    start_urls = ['https://www.example.com/']
    allowed_domains = ['example.com']
    visited_urls = set()  # 避免重复爬取

    def start_requests(self):
        for url in self.start_urls:
            print("+++++++++++++++++++++++++++++++++++++++++++++++++++++",url)
            self.visited_urls.add(url)
            yield SeleniumRequest(url=url, callback=self.parse,dont_filter=True)

    def parse(self, response):
        print(response.url)
        item = {'url': response.url, 'html': response.body}
        yield item

        # 提取当前页面所有符合域名的链接
        link_extractor = LinkExtractor(allow_domains=self.allowed_domains)
        for link in link_extractor.extract_links(response):
            if link.url not in self.visited_urls:
                self.visited_urls.add(link.url)
                # 生成SeleniumRequest确保JS渲染
                yield SeleniumRequest(url=link.url, callback=self.parse, dont_filter=True)

    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'scrapy_selenium.SeleniumMiddleware': 800
        },
        'SELENIUM_DRIVER_NAME': 'chrome',
        'SELENIUM_DRIVER_EXECUTABLE_PATH': '/home/ubuntu/selenium_drivers/chromedriver',
        'SELENIUM_DRIVER_ARGUMENTS': ['-headless']
    }

• 确认ChromeDriver与Chrome浏览器版本完全匹配，版本不兼容会导致Selenium无法正常提取页面链接。
• 如果页面链接是滚动/点击加载的，需在SeleniumRequest中添加等待条件：

  yield SeleniumRequest(
      url=url,
      callback=self.parse,
      dont_filter=True,
      wait_until=lambda driver: driver.execute_script('return document.readyState') == 'complete'
  )
  

内容的提问来源于stack exchange，提问作者Hans M