Python HTTPS网页爬取遇SSL证书PEM库错误求助

问题详情 #

我尝试用Python的requests-html库爬取网站https://icp.administracionelectronica.gob.es/icpplus/index.html，代码如下：

from requests_html import HTMLSession

headers={
  'Host':'icp.administracionelectronica.gob.es',
  'Cookie':'viewed_cookie_policy=yes',
  'Cache-Control':'max-age=0',
  'Sec-Ch-Ua':'"Not.A/Brand";v="8", "Chromium";v="114"',
  'Sec-Ch-Ua-Mobile':'?0',
  'Sec-Ch-Ua-Platform':'"Linux"',
  'Upgrade-Insecure-Requests':'1',
  'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
  'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
  'Sec-Fetch-Site':'none',
  'Sec-Fetch-Mode':'navigate',
  'Sec-Fetch-User':'?1',
  'Sec-Fetch-Dest':'document',
  'Accept-Encoding':'gzip, deflate',
  'Accept-Language':'en-US,en;q=0.9',
  'Connection':'close'
}

session = HTMLSession()

r = session.get('https://icp.administracionelectronica.gob.es/icpplus/index.html', headers=headers, cert='/home/kali/Documents/Jorge/administracionelectronica-gob-es-chain.pem')

r.html.render()

已通过Firefox下载该网站的3个证书链并合并为一个包含9个证书的PEM文件，格式示例：

-----BEGIN CERTIFICATE-----
MIIGtzCCBZ+gAwIBAgIQC0CzDw53M/lN743NvBzAyDANBgkqhkiG9w0BAQsFADBe
[...]
N9s6FHkEzLhRQXEK3r/q1H2MEt1DST7LyHy2jRaRwRE/fCYgOr3SiZy+4Q==
-----END CERTIFICATE-----
[...]
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
[...]
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----

运行代码时出现以下错误：

File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='icp.administracionelectronica.gob.es', port=443): Max retries exceeded with url: /icpplus/index.html (Caused by SSLError(SSLError(524297, '[SSL] PEM lib (_ssl.c:3921)')))

File "/usr/lib/python3/dist-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='icp.administracionelectronica.gob.es', port=443): Max retries exceeded with url: /icpplus/index.html (Caused by SSLError(SSLError(524297, '[SSL] PEM lib (_ssl.c:3921)')))

解决步骤 #

• 修正PEM文件格式
  你提供的证书示例中，-----BEGIN CERTIFICATE-----行有前置空格，这会导致SSL库解析失败。必须确保所有证书的起始、结束标记顶格无空格，正确格式如下：

  -----BEGIN CERTIFICATE-----
  MIIGtzCCBZ+gAwIBAgIQC0CzDw53M/lN743NvBzAyDANBgkqhkiG9w0BAQsFADBe
  [...]
  N9s6FHkEzLhRQXEK3r/q1H2MEt1DST7LyHy2jRaRwRE/fCYgOr3SiZy+4Q==
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
  [...]
  CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
  -----END CERTIFICATE-----
  

  打开你的PEM文件，删除所有标记行的前置空格后保存重试。

• 替换参数用途
  requests的cert参数用于指定客户端证书，而验证服务器证书链需要使用verify参数。修改代码中的请求行：

  r = session.get('https://icp.administracionelectronica.gob.es/icpplus/index.html', headers=headers, verify='/home/kali/Documents/Jorge/administracionelectronica-gob-es-chain.pem')
  

• 简化测试流程
  先禁用r.html.render()，测试基础SSL连接是否正常，确认证书问题解决后再启用渲染功能：

  from requests_html import HTMLSession
  
  headers={
    'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
    'Cookie':'viewed_cookie_policy=yes'
  }
  
  session = HTMLSession()
  r = session.get('https://icp.administracionelectronica.gob.es/icpplus/index.html', headers=headers, verify='/path/to/your/corrected.pem')
  print(r.status_code)
  print(r.text[:500])
  

• 备选：使用系统信任库
  如果上述方法无效，可以尝试依赖系统默认SSL证书库，直接移除verify参数（或设为True）：

  r = session.get('https://icp.administracionelectronica.gob.es/icpplus/index.html', headers=headers)
  

内容的提问来源于stack exchange，提问作者Antonio Criado