Python中cloudscraper禁用安全证书校验报错的解决方法

我使用cloudscraper绕过部分网站的Cloudflare WAF原本运行正常，但添加verify=False参数禁用安全证书校验时，抛出如下错误：

Traceback (most recent call last):
  File "x:\documents\python\django_projects\storage\saver\image_save.py", line 49, in fetch
    response  = session.request(method=method,url=url,headers=headers,params=params,timeout=timeout,verify=False)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\cloudscraper\__init__.py", line 257, in request
    self.perform_request(method, url, *args, **kwargs)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\cloudscraper\__init__.py", line 190, in perform_request 
    return super(CloudScraper, self).request(method, url, *args, **kwargs)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen        
    httplib_response = self._make_request(
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 382, in _make_request  
    self._validate_conn(conn)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 398, in connect
    context.verify_mode = resolve_cert_reqs(self.cert_reqs)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 738, in verify_mode
    super(SSLContext, SSLContext).verify_mode.__set__(self, value)
ValueError: Cannot set verify_mode to CERT_NONE when check_hostname is enabled.

我的原始代码如下：

import cloudscraper
from urllib.parse import urlparse

session = cloudscraper.CloudScraper(
            browser={
            'browser': 'chrome',
            'platform': 'windows',
            'mobile': False
                    },)
def fetch(url : str , method : str = "GET",refer : str = None,params : dict = {},
          timeout : int = 40) -> bytes:
    
    url_r  = urlparse(url)
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
        'referer': refer if refer else "%s://%s/"%(url_r.scheme,url_r.netloc)}
    try:
        response  = session.request(method=method,url=url,headers=headers,params=params,timeout=timeout,verify=False)
        return response.content 
    except  Exception as e:   print(e)

解决方案 #

这个错误的核心原因是：SSL上下文同时启用了check_hostname和verify_mode=CERT_NONE，这两者存在冲突——当关闭证书校验时，不能同时开启主机名检查。以下是两种可行的解决办法：

方法1：自定义SSL适配器（推荐，精准控制） #

给cloudscraper的session挂载自定义的SSL适配器，同时关闭check_hostname并设置verify_mode=CERT_NONE，避免冲突：

import cloudscraper
import ssl
from urllib.parse import urlparse
from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager

# 自定义适配无证书校验的SSL适配器
class InsecureSSLAdapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        # 创建SSL上下文并关闭主机名检查和证书校验
        ssl_context = ssl.create_default_context()
        ssl_context.check_hostname = False
        ssl_context.verify_mode = ssl.CERT_NONE
        kwargs['ssl_context'] = ssl_context
        super().init_poolmanager(*args, **kwargs)

# 初始化cloudscraper会话
session = cloudscraper.CloudScraper(
    browser={
        'browser': 'chrome',
        'platform': 'windows',
        'mobile': False
    },
)
# 给HTTPS请求挂载自定义适配器
session.mount('https://', InsecureSSLAdapter())

def fetch(url : str , method : str = "GET",refer : str = None,params : dict = {},
          timeout : int = 40) -> bytes:
    
    url_r  = urlparse(url)
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
        'referer': refer if refer else "%s://%s/"%(url_r.scheme,url_r.netloc)}
    try:
        # 此处无需再传verify=False，适配器已处理
        response  = session.request(method=method,url=url,headers=headers,params=params,timeout=timeout)
        return response.content 
    except Exception as e:
        print(e)

方法2：全局禁用SSL校验（不推荐，影响范围广） #

在代码开头添加全局设置，强制关闭所有请求的SSL证书校验和主机名检查，适合测试环境快速验证：

import ssl
import urllib3

# 关闭urllib3的不安全请求警告
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# 全局使用未验证的SSL上下文
ssl._create_default_https_context = ssl._create_unverified_context

添加后，原代码中verify=False的设置即可正常生效，但这种方法会影响当前Python进程内的所有HTTPS请求，安全性较低，不建议在生产环境使用。

内容的提问来源于stack exchange，提问作者urek mazino