Scrapy-Redis爬虫频繁卡在Starting new HTTPS connection问题求助

针对Scrapy-Redis批量爬取时频繁卡在DEBUG: Starting new HTTPS connection的问题，可通过以下配置调整解决：

1. 设置连接超时与重试机制 #

避免爬虫因等待连接建立超时挂起，同时对失败请求进行重试：

# settings.py
DOWNLOAD_TIMEOUT = 15  # 连接超时时间，单位秒
RETRY_ENABLED = True
RETRY_TIMES = 3  # 重试次数
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]  # 需要重试的状态码

2. 优化连接池与请求并发 #

减少频繁新建连接，控制单域名并发数，避免触发目标网站限流：

# settings.py
CONCURRENT_REQUESTS = 8  # 全局最大并发请求数
CONCURRENT_REQUESTS_PER_DOMAIN = 2  # 单域名并发请求数
DOWNLOAD_DELAY = 1  # 请求间隔，单位秒

# 开启HTTP持久连接，复用已有连接减少新建开销
DOWNLOAD_HANDLERS = {
    "http": "scrapy.core.downloader.handlers.http.HttpDownloadHandler",
    "https": "scrapy.core.downloader.handlers.http.HttpDownloadHandler",
}

3. 调整DNS解析配置 #

避免DNS解析卡住导致连接建立失败：

# settings.py
DNS_TIMEOUT = 10  # DNS解析超时时间
# 或禁用Scrapy默认DNS缓存，使用系统缓存
DNS_ENABLED = False

4. 清理Scrapy-Redis调度队列 #

避免队列中堆积无效请求导致爬虫反复尝试：

# settings.py
SCHEDULER_PERSIST = False  # 爬虫停止时自动清空Redis队列，避免残留问题URL

5. 应对目标网站反爬 #

通过更换User-Agent、使用代理IP分散请求来源，降低被限流概率：

# settings.py - 配置User-Agent池
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}

相关日志：
2022-11-29 10:29:37 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): learn.microsoft.com:443
2022-11-29 10:29:38 [urllib3.connectionpool] DEBUG: https://learn.microsoft.com:443 "GET /es-es/dotnet/api/system.drawing.configuration.systemdrawingsection.bitmapsuffix?view=netframework-4.8 HTTP/1.1" 200 10906
2022-11-29 10:29:38 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): experienceleaguecommunities.adobe.com:443
2022-11-29 10:29:40 [urllib3.connectionpool] DEBUG: https://experienceleaguecommunities.adobe.com:443 "GET /t5/workfront-questions/linking-connecting-projects/m-p/472753/highlight/true HTTP/1.1" 200 40251
2022-11-29 10:29:40 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): experienceleaguecommunities.adobe.com:443
2022-11-29 10:30:13 [urllib3.connectionpool] DEBUG: https://experienceleaguecommunities.adobe.com:443 "GET /t5/workfront-questions/linking-connecting-projects/m-p/472752/highlight/true HTTP/1.1" 200 40239
2022-11-29 10:30:14 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): learn.microsoft.com:443
2022-11-29 10:30:15 [urllib3.connectionpool] DEBUG: https://learn.microsoft.com:443 "GET /es-es/dotnet/api/system.windows.controls.primitives.selector.selectionchanged?view=windowsdesktop-7.0 HTTP/1.1" 200 10895
2022-11-29 10:30:15 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): learn.microsoft.com:443
2022-11-29 10:30:15 [urllib3.connectionpool] DEBUG: https://learn.microsoft.com:443 "GET /zh-cn/dotnet/api/system.drawing.configuration.systemdrawingsection.bitmapsuffix?view=netframework-4.8 HTTP/1.1" 200 11248
2022-11-29 10:30:15 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): learn.microsoft.com:443

内容的提问来源于stack exchange，提问作者chester