Python批量下载网站PDF时触发SSL证书验证失败报错排查

在Spyder环境中编写Python爬虫脚本，目标是爬取Sedar平台指定页面（https://sedar.com/DisplayCompanyDocuments.do?lang=EN&issuerNo=00030202）下的所有PDF文件，存储到本地指定目录。脚本已导入os、requests、ssl、BeautifulSoup等依赖，完成了请求头配置、本地存储目录自动创建、页面链接解析、文件下载写入逻辑的编写，但运行时首次发起requests.get请求就抛出异常，爬取流程中断。

import os
import requests
import ssl
from urllib.parse import urljoin
from bs4 import BeautifulSoup

url = "https://sedar.com/DisplayCompanyDocuments.do?lang=EN&issuerNo=00030202"
headers = {
'Host': 'sedar.com',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-User': '?1',
'TE': 'trailers'
 }

# 不存在目标文件夹则自动创建
folder_location = r'C:\Users\jay_patel1\...\test'
base_url = 'https://sedar.com'
if not os.path.exists(folder_location):
    os.mkdir(folder_location)

response = requests.get(url=url, headers=headers)

soup= BeautifulSoup(response.text, "html.parser")

forms = soup.find_all('form')
print(len(forms))
counter = 0
for form in forms:
    action = form['action']
    doc_link = base_url+action
    filename = os.path.join(folder_location,str(counter)+'.pdf')
    counter = counter + 1
    headers['Cookie'] = '__uzma=28481475-7c5c-4e62-ad08-ebabddb1fff0; __uzmb=1657722275; __uzme=5741; __uzmc=7155215141911; __uzmd=1657726129; TS015c16dc=016abe8a18b65821e9668faf87606307d873066e84546d78152ebf0e1734d6f553b800143820e548c5ead2bd19b0cd0d8c6d8b4fc0937d7b83d0619de26e621bf2db8d6741; __ssds=2; __ssuzjsr2=a9be0cd8e; __uzmaj2=5c5b0a58-f490-41e6-945d-6a4766be7d82; __uzmbj2=1657722277; __uzmcj2=158447041868; __uzmdj2=1657726118; JSESSIONID=0000Hh7sa7ec87A8vlJsaqRiHhX:1884ter20'
    with open(filename, 'wb') as f :
        print("Writing")
        print(doc_link)
        content = requests.get(url=doc_link, headers=headers).content
        f.write(content)
        f.close()

报错核心为SSLCertVerificationError，提示证书验证失败、无法获取本地颁发者证书，最终触发HTTPS连接最大重试超限错误，完整报错栈如下：

Traceback (most recent call last):

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 1010, in _validate_conn
    conn.connect()

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\connection.py", line 416, in connect
    self.sock = ssl_wrap_socket(

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\util\ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)

  File "C:\Users\jay_patel1\Anaconda3\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(

  File "C:\Users\jay_patel1\Anaconda3\lib\ssl.py", line 1040, in _create
    self.do_handshake()

  File "C:\Users\jay_patel1\Anaconda3\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))

MaxRetryError: HTTPSConnectionPool(host='sedar.com', port=443): Max retries exceeded with url: /DisplayCompanyDocuments.do?lang=EN&issuerNo=00030202 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\jay_patel1\Downloads\pdf download (1).py", line 36, in <module>
    response = requests.get(url=url, headers=headers)

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)

  File "C:\Users\jay_patel1\Anaconda3\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)

SSLError: HTTPSConnectionPool(host='sedar.com', port=443): Max retries exceeded with url: /DisplayCompanyDocuments.do?lang=EN&issuerNo=00030202 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

根因说明 #

仅导入ssl模块不会自动修改requests的SSL校验逻辑，requests默认会强制校验目标站点的SSL证书有效性，本地Python环境缺失sedar.com对应的根证书链时，就会抛出证书验证失败错误，和是否导入ssl模块没有直接关系。
另外原代码还存在两个稳定性问题：

• 硬编码了Cookie值，Cookie存在会话有效期，过期后会导致文件下载失败
• 直接用字符串拼接文档链接，容易出现路径斜杠重复/缺失的问题，之前导入的urljoin没有被使用

修复方案 #

方案1：快速调试修复（临时关闭SSL校验，仅适合本地测试场景） #

• 所有requests.get()调用位置添加verify=False参数，关闭证书校验
• 导入urllib3模块关闭证书校验的警告提示，避免控制台输出冗余信息
• 改用requests.Session()自动维护会话Cookie，不需要手动硬编码Cookie值
• 用urljoin拼接文档链接，避免路径错误

关键修改点示例：

# 新增导入
import urllib3
# 关闭SSL未验证警告
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# 用Session自动维护Cookie
session = requests.Session()
session.headers.update(headers)

# 首次请求添加verify=False
response = session.get(url=url, verify=False)

# 文档链接改用urljoin拼接
doc_link = urljoin(base_url, action)

# 下载请求同样添加verify=False
content = session.get(url=doc_link, verify=False).content

方案2：规范修复（保留SSL校验，无安全风险） #

1. 先升级本地证书依赖包到最新版本，在终端执行命令：
   pip install --upgrade certifi
2. 升级完成后重新运行脚本，大部分证书校验问题会自动解决
3. 如果仍报错，可以手动从浏览器导出sedar.com的根证书文件，在请求时通过verify参数指定证书路径，例如verify="./sedar_root.cer"即可正常校验。

内容的提问来源于stack exchange，提问作者Jay Patel