Scrapy调用API遇SSL错误：unexpected eof while reading求助

问题场景 #

使用Scrapy向https://www.macmap.org/api/v2/ntlc-products发送GET请求时，遭遇SSL错误，报错为unexpected eof while reading，重试3次后仍无法获取响应。

请求代码 #

url = 'https://www.macmap.org/api/v2/ntlc-products?countryCode=764&level=8&code=010229'

headers = {
    "Accept": "application/json, text/javascript, */*; q=0.01",
    "Accept-Language": "en-US,en;q=0.9,ml;q=0.8",
    "Connection": "keep-alive",
    "Content-Type": "application/json; charset=utf-8",
    "DNT": "1",
    "Referer": "https://www.macmap.org/en//query/results?reporter=764&partner=004&product=010229&level=6",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-origin",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest",
}


request = Request(
    url=url,
    method='GET',
    dont_filter=True,
    headers=headers,
)

fetch(request)

报错信息 #

2024-07-17 06:07:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.macmap.org/api/v2/ntlc-products?countryCode=764&level=8&code=010229> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unexpected eof while reading')]>]
2024-07-17 06:07:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.macmap.org/api/v2/ntlc-products?countryCode=764&level=8&code=010229> (failed 2 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unexpected eof while reading')]>]
2024-07-17 06:07:03 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.macmap.org/api/v2/ntlc-products?countryCode=764&level=8&code=010229> (failed 3 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unexpected eof while reading')]>]
---------------------------------------------------------------------------
ResponseNeverReceived                     Traceback (most recent call last)
Cell In[1], line 27
      5 headers = {
      6     "Accept": "application/json, text/javascript, */*; q=0.01",
      7     "Accept-Language": "en-US,en;q=0.9,ml;q=0.8",
   (...)
     16     "X-Requested-With": "XMLHttpRequest",
     17 }
     20 request = Request(
     21     url=url,
     22     method='GET',
     23     dont_filter=True,
     24     headers=headers,
     25 )
---&gt; 27 fetch(request)

File /opt/venv-python3.9-scrapy/lib/python3.9/site-packages/scrapy/shell.py:110, in Shell.fetch(self, request_or_url, spider, redirect, **kwargs)
    108 response = None
    109 try:
--&gt; 110     response, spider = threads.blockingCallFromThread(
    111         reactor, self._schedule, request, spider)
    112 except IgnoreRequest:
    113     pass

File /opt/venv-python3.9-scrapy/lib/python3.9/site-packages/twisted/internet/threads.py:119, in blockingCallFromThread(reactor, f, *a, **kw)
    117 result = queue.get()
    118 if isinstance(result, failure.Failure):
--&gt; 119     result.raiseException()
    120 return result

File /opt/venv-python3.9-scrapy/lib/python3.9/site-packages/twisted/python/failure.py:475, in Failure.raiseException(self)
    474 def raiseException(self):
--&gt; 475     raise self.value.with_traceback(self.tb)

ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unexpected eof while reading')]>]

环境版本 #

• Scrapy==2.5.0
• pyOpenSSL==22.0.0
• Cryptography==38.0.4

解决方案 #

• 更新依赖包：当前使用的Scrapy和SSL相关库版本较旧，对现代SSL协议支持不足。执行以下命令升级到兼容版本：

  pip install --upgrade scrapy pyopenssl cryptography
  

  建议升级到Scrapy 2.10+、pyOpenSSL 24.0+、cryptography 42+版本。

• 调整SSL协议配置：在Scrapy项目的settings.py中添加以下配置，强制使用TLS 1.2协议，同时开启SSL日志排查细节：

  DOWNLOADER_CLIENT_TLS_METHOD = 'TLSv1.2'
  DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING = True
  

• 临时禁用SSL验证（仅测试用）：若上述方法无效，可临时关闭SSL证书验证（生产环境不推荐，存在安全风险）：

  DOWNLOADER_CLIENT_TLS_VERIFY = False
  

• 验证目标网站SSL状态：先用curl -v https://www.macmap.org测试目标网站的SSL连接是否正常，确认是否是网站端的SSL配置问题。

内容的提问来源于stack exchange，提问作者Jff