多线程环境下Scrapy爬取失效问题求助

问题根源分析 #

1. CrawlerProcess.start()是阻塞调用，会直接卡住当前线程，导致循环无法持续执行；且Twisted Reactor在子线程中重复启动/销毁的操作极易引发兼容性问题。
2. 全局变量list_of_new_hrefs未做线程同步处理，多线程读写时会出现数据竞争，导致获取的URL列表异常。
3. 循环内重复删除twisted.internet.reactor模块的做法不可靠，容易触发未知错误。

修正方案与代码实现 #

1. 导入必要模块

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from twisted.internet import reactor
import threading
import time
from scrapy.utils.project import get_project_settings

2. 定义线程安全的全局变量与锁

list_lock = threading.Lock()
list_of_new_hrefs = []

3. 重写爬取线程逻辑

def crawler_runner(barrier):
    global list_of_new_hrefs

    # 一次性配置日志与Scrapy设置，避免重复初始化
    configure_logging({'LOG_LEVEL': 'CRITICAL'})
    s = get_project_settings()
    s['USER_AGENT'] = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
    s['CONCURRENT_REQUESTS_PER_IP'] = 30
    s['DOWNLOADER_MIDDLEWARES'] = {
        'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
        'divar_crawler.middlewares.TooManyRequestsRetryMiddleware': 543,
        'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
        'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
        'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,
    }
    runner = CrawlerRunner(s)

    def crawl_batch(batch_urls):
        # 启动爬取任务，完成后停止Reactor
        deferred = runner.crawl(QuotesSpider, start_urls=batch_urls)
        deferred.addBoth(lambda _: reactor.stop())
        reactor.run(installSignalHandlers=False)  # 避免子线程信号处理冲突

    while True:
        batch_urls = []
        # 加锁批量获取URL，解决多线程数据竞争
        with list_lock:
            if len(list_of_new_hrefs) > 0:
                batch_size = min(100, len(list_of_new_hrefs))
                batch_urls = list_of_new_hrefs[:batch_size]
                del list_of_new_hrefs[:batch_size]  # 移除已分配的URL，避免重复爬取

        if not batch_urls:
            print('Nothing to crawl...\n')
            time.sleep(2)
            continue

        # 启动独立子线程执行爬取，避免阻塞主循环
        crawl_thread = threading.Thread(target=crawl_batch, args=(batch_urls,))
        crawl_thread.start()
        crawl_thread.join()  # 等待当前批次爬取完成后再处理下一批

        time.sleep(2)

关键优化点 #

• 用CrawlerRunner替代CrawlerProcess，更灵活控制Reactor的生命周期，适配子线程环境。
• 给全局URL列表添加线程锁，确保多线程读写时的数据一致性。
• 每次爬取后从全局列表中移除已处理的URL，避免重复爬取。
• 将单批次爬取逻辑封装到独立子线程，避免阻塞主循环的URL监听逻辑。

内容的提问来源于stack exchange，提问作者kian kian