如何使用Scrapy按顺序爬取多URL？解决Reactor不可重启错误

错误原因 #

你遇到的ReactorNotRestartable错误，核心原因是Twisted的事件循环（reactor）只能被启动一次。你的原有代码每次调用do_crawl都会创建新的CrawlerProcess并调用process.start()，第二次调用时reactor已经处于启动状态，无法重复启动，因此触发报错。

解决方法 #

方法1：单进程批量添加爬虫，单次启动reactor #

将所有爬取任务添加到同一个CrawlerProcess实例中，仅启动一次reactor即可避免错误。若需要严格控制请求顺序，可配合Scrapy的并发限制参数。

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    name = "myspider"

    def __init__(self, start_urls=None, allowed_domains=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.start_urls = start_urls or []
        self.allowed_domains = allowed_domains or []

    def parse(self, response):
        data = {
            "url": response.url,
            "title": response.xpath("//title/text()").get(),
        }
        yield data

def main():
    process = CrawlerProcess(settings={
        "FEEDS": {
            "results.json": {"format": "json", "overwrite": False},  # 追加写入结果
        },
        "CONCURRENT_REQUESTS": 1,  # 限制并发请求数为1，确保顺序处理
    })

    # 逐个添加爬虫实例
    process.crawl(MySpider, 
                  start_urls=["https://www.wikipedia.org/"], 
                  allowed_domains=["wikipedia.org"])
    process.crawl(MySpider, 
                  start_urls=["https://www.stackoverflow.com/"], 
                  allowed_domains=["stackoverflow.com"])

    # 仅启动一次reactor
    process.start()

if __name__ == "__main__":
    main()

注：通过给Spider添加__init__方法，用实例参数传递URL和域名，避免修改类属性导致的全局共享问题。

方法2：用Twisted异步回调实现严格顺序爬取 #

如果需要确保前一个域名的爬取任务完全结束后，再启动下一个任务，可以利用Twisted的defer机制实现链式执行：

import scrapy
from scrapy.crawler import CrawlerProcess
from twisted.internet.defer import inlineCallbacks

class MySpider(scrapy.Spider):
    name = "myspider"

    def __init__(self, start_urls=None, allowed_domains=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.start_urls = start_urls or []
        self.allowed_domains = allowed_domains or []

    def parse(self, response):
        data = {
            "url": response.url,
            "title": response.xpath("//title/text()").get(),
        }
        yield data

@inlineCallbacks
def crawl_sequentially():
    process = CrawlerProcess(settings={
        "FEEDS": {
            "results.json": {"format": "json", "overwrite": False},
        },
    })

    # 等待第一个爬取任务完成
    yield process.crawl(MySpider, 
                        start_urls=["https://www.wikipedia.org/"], 
                        allowed_domains=["wikipedia.org"])
    # 第一个任务完成后，启动第二个
    yield process.crawl(MySpider, 
                        start_urls=["https://www.stackoverflow.com/"], 
                        allowed_domains=["stackoverflow.com"])

    # 启动reactor直到所有任务结束
    process.start()

if __name__ == "__main__":
    crawl_sequentially()

方法3：多进程隔离reactor #

利用Python多进程，让每个爬取任务在独立进程中运行，每个进程拥有自己的reactor，彻底避免重启冲突：

import scrapy
from scrapy.crawler import CrawlerProcess
import multiprocessing

class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = []
    allowed_domains = []

    def parse(self, response):
        data = {
            "url": response.url,
            "title": response.xpath("//title/text()").get(),
        }
        yield data

def do_crawl(start_url, allowed_domain):
    spider = MySpider
    spider.start_urls = [start_url]
    spider.allowed_domains = [allowed_domain]

    process = CrawlerProcess(settings={
        "FEEDS": {
            "results.json": {"format": "json", "overwrite": False},
        },
    })

    process.crawl(spider)
    process.start()

if __name__ == "__main__":
    # 创建独立进程执行爬取任务
    p1 = multiprocessing.Process(target=do_crawl, args=("https://www.wikipedia.org/", "wikipedia.org"))
    p2 = multiprocessing.Process(target=do_crawl, args=("https://www.stackoverflow.com/", "stackoverflow.com"))

    # 按顺序启动并等待进程完成
    p1.start()
    p1.join()
    p2.start()
    p2.join()

内容的提问来源于stack exchange，提问作者Amir H