Scrapy宽爬未完成全部起始请求即终止的问题求助

问题背景 #

使用Scrapy执行百万级网站的宽爬任务，已连接PostgreSQL数据库记录爬取状态。本地Windows 11机器运行正常，但在公司Azure Windows 10服务器上，爬虫会提前终止，仅爬取数百到数千个URL（数量不固定），日志无明确错误提示，finish_reason显示为finished，但仍有大量未处理URL。

核心代码片段 #

加载未处理URL的数据库查询代码 #

def get_unprocessed_urls(self, suffix):
    """
    Fetch unprocessed urls.
    """

    print(f'Fetching unprocessed urls for suffix {suffix}...')

    cursor = self.connection.cursor('unprocessed_urls_cursor', withhold=True)
    cursor.itersize = 1000
    cursor.execute(f"""
        SELECT su.id, su.url FROM seed_url su
        LEFT JOIN footer_seed_url_status fsus ON su.id = fsus.seed_url_id
        WHERE su.url LIKE '%.{suffix}' AND fsus.seed_url_id IS NULL;
    """)

    ID = 0
    URL = 1

    urls = [Url(url_row[ID], self.validate_url(url_row[URL])) for url_row in cursor]

    print('len urls:', len(urls))
    return urls

Spider核心代码 #

class FooterSpider(scrapy.Spider):

    ...

    def start_requests(self):
        urls = self.handler.get_unprocessed_urls(self.suffix)
        for url in urls:
            yield scrapy.Request(
                url=url.url,
                callback=self.parse,
                errback=self.errback,
                meta={
                    'seed_url_id': url.id,
                }
            )

    def parse(self, response):
        try:
            seed_url_id = response.meta.get('seed_url_id')
            print(response.url)
            soup = BeautifulSoup(response.text, 'html.parser')
            footer = soup.find('footer')
            item = FooterItem(
                seed_url_id=seed_url_id,
                html=str(footer) if footer is not None else None,
                url=response.url
            )
            yield item
            print(f'Successfully processed url {response.url}')
        except Exception as e:
            print('Error while processing url', response.url)
            print(e)
            seed_url_id = response.meta.get('seed_url_id')
            cursor  = self.handler.connection.cursor()
            cursor.execute(
                "INSERT INTO footer_seed_url_status(seed_url_id, status) VALUES(%s, %s)",
                (seed_url_id, str(e)))
            self.handler.connection.commit()

    def errback(self, failure):
        print(failure.value)
        try:
            error = repr(failure.value)
            request = failure.request
            seed_url_id = request.meta.get('seed_url_id')
            cursor  = self.handler.connection.cursor()
            cursor.execute(
                "INSERT INTO footer_seed_url_status(seed_url_id, status) VALUES(%s, %s)",
                (seed_url_id, error))
            self.handler.connection.commit()
        except Exception as e:
            print(e)

爬取配置 #

SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'
CONCURRENT_REQUESTS = 100
CONCURRENT_ITEMS=1000
SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'
REACTOR_THREADPOOL_MAXSIZE = 20
COOKIES_ENABLED = False
DOWNLOAD_DELAY = 0.2

问题现象示例日志 #

{'downloader/exception_count': 2,
 'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 1,
 'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 1,
 'downloader/request_bytes': 345073,
 'downloader/request_count': 1481,
 'downloader/request_method_count/GET': 1481,
 'downloader/response_bytes': 1977255,
 'downloader/response_count': 1479,
 'downloader/response_status_count/200': 46,
 'downloader/response_status_count/301': 791,
 'downloader/response_status_count/302': 512,
 'downloader/response_status_count/303': 104,
 'downloader/response_status_count/308': 2,
 'downloader/response_status_count/403': 2,
 'downloader/response_status_count/404': 22,
 'dupefilter/filtered': 64,
 'elapsed_time_seconds': 113.895788,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2023, 8, 3, 11, 46, 31, 889491),
 'httpcompression/response_bytes': 136378,
 'httpcompression/response_count': 46,
 'log_count/ERROR': 3,
 'log_count/INFO': 11,
 'log_count/WARNING': 7,
 'response_received_count': 43,
 "robotstxt/exception_count/<class 'twisted.internet.error.DNSLookupError'>": 1,
 "robotstxt/exception_count/<class 'twisted.web._newclient.ResponseNeverReceived'>": 1,
 'robotstxt/request_count': 105,
 'robotstxt/response_count': 43,
 'robotstxt/response_status_count/200': 21,
 'robotstxt/response_status_count/403': 2,
 'robotstxt/response_status_count/404': 20,
 'scheduler/dequeued': 151,
 'scheduler/dequeued/memory': 151,
 'scheduler/enqueued': 151,
 'scheduler/enqueued/memory': 151,
 'start_time': datetime.datetime(2023, 8, 3, 11, 44, 37, 993703)}
2023-08-03 11:46:31 [scrapy.core.engine] INFO: Spider closed (finished)

排查方向与解决方法 #

1. 内存过载：一次性加载百万URL到内存 #

当前代码将所有未处理URL一次性存入列表urls = [Url(...) for url_row in cursor]，百万级URL会占用大量内存，Azure服务器内存资源有限时，会导致进程被系统强制终止或Scrapy调度器异常。

修改方案：改用生成器分批加载URL，避免内存爆炸：

def get_unprocessed_urls(self, suffix):
    print(f'Fetching unprocessed urls for suffix {suffix}...')
    cursor = self.connection.cursor('unprocessed_urls_cursor', withhold=True)
    cursor.itersize = 1000
    cursor.execute(f"""
        SELECT su.id, su.url FROM seed_url su
        LEFT JOIN footer_seed_url_status fsus ON su.id = fsus.seed_url_id
        WHERE su.url LIKE '%.{suffix}' AND fsus.seed_url_id IS NULL;
    """)
    ID = 0
    URL = 1
    # 返回生成器，逐个生成URL对象
    for url_row in cursor:
        yield Url(url_row[ID], self.validate_url(url_row[URL]))

对应start_requests遍历生成器：

def start_requests(self):
    # 直接遍历生成器，无需一次性加载所有URL
    for url in self.handler.get_unprocessed_urls(self.suffix):
        yield scrapy.Request(
            url=url.url,
            callback=self.parse,
            errback=self.errback,
            meta={'seed_url_id': url.id}
        )

2. 调度器配置失效：磁盘队列未生效 #

配置中指定了磁盘队列，但未明确设置SCHEDULER，可能导致Scrapy默认使用内存队列，百万级请求撑爆内存后爬虫终止。另外Azure服务器可能限制磁盘写入权限，导致磁盘队列无法持久化请求。

修改方案：

• 明确启用调度器并配置任务目录，持久化爬虫状态：

SCHEDULER = 'scrapy.core.scheduler.Scheduler'
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'
# 添加任务目录，用于保存爬虫状态和磁盘队列
JOBDIR = './scrapy_broad_crawl_job'
# 其他配置保持不变
CONCURRENT_REQUESTS = 100
CONCURRENT_ITEMS=1000
SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'
REACTOR_THREADPOOL_MAXSIZE = 20
COOKIES_ENABLED = False
DOWNLOAD_DELAY = 0.2

• 确保Azure服务器上JOBDIR目录有写入权限，可手动创建目录并设置权限。

3. 网络/DNS限制：Azure服务器网络限流 #

日志中出现DNSLookupError，说明Azure服务器可能存在DNS查询限流或出站请求限制，导致大量请求无法发出，Scrapy认为队列空而终止。

修改方案：

• 配置DNS缓存，安装scrapy-dns-resolver插件，减少DNS查询次数：

pip install scrapy-dns-resolver

在配置中添加：

DNS_RESOLVER = 'scrapy_dns_resolver.CachedDNSResolver'
DNS_CACHE_SIZE = 10000

• 检查Azure服务器的网络安全组（NSG）规则，确保出站HTTP/HTTPS请求未被限制；增加DNS超时时间：

DOWNLOAD_TIMEOUT = 30
DNS_TIMEOUT = 10

4. 数据库连接泄漏：重复创建游标未关闭 #

在parse和errback方法中，每次创建游标但未关闭，会导致数据库连接泄漏，Azure服务器上数据库连接数耗尽后，爬虫无法更新状态，甚至卡住终止。

修改方案：使用上下文管理器自动关闭游标：

# parse方法异常处理部分
try:
    # ... 原有逻辑 ...
except Exception as e:
    self.logger.error(f"Error processing {response.url}: {str(e)}")
    seed_url_id = response.meta.get('seed_url_id')
    # 使用上下文管理器，自动关闭游标
    with self.handler.connection.cursor() as cursor:
        cursor.execute(
            "INSERT INTO footer_seed_url_status(seed_url_id, status) VALUES(%s, %s)",
            (seed_url_id, str(e))
        )
    self.handler.connection.commit()

5. Windows平台输出阻塞：大量print导致进程挂起 #

Windows控制台输出缓冲区有限，大量print语句会导致输出阻塞，进程无法继续执行，最终看似“提前终止”。

修改方案：替换print为Scrapy内置日志系统：

def parse(self, response):
    try:
        seed_url_id = response.meta.get('seed_url_id')
        self.logger.info(f"Processing URL: {response.url}")
        soup = BeautifulSoup(response.text, 'html.parser')
        footer = soup.find('footer')
        item = FooterItem(
            seed_url_id=seed_url_id,
            html=str(footer) if footer is not None else None,
            url=response.url
        )
        yield item
        self.logger.info(f"Successfully processed URL: {response.url}")
    except Exception as e:
        self.logger.error(f"Error processing {response.url}: {str(e)}")
        # ... 数据库更新逻辑 ...

6. 爬虫终止条件验证：确认请求是否被正确调度 #

检查Scrapy的去重机制是否过滤了大量请求，或3xx重定向后的URL被误判为重复。可临时关闭去重测试：

DUPEFILTER_CLASS = 'scrapy.dupefilters.BaseDupeFilter'

测试后再恢复，同时确保validate_url方法不会生成重复URL。

内容的提问来源于stack exchange，提问作者druskacik