使用Airflow及Python Requests调用POST API遇SSL证书验证失败求助

使用Airflow的HttpOperator调用API时遇到以下SSL错误：

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<api host>', port=443): Max retries exceeded with url: <endpoint path> (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

随后参考HttpHook编写了Requests测试代码，但仍出现相同错误：

import requests

data = {}
data['subject'] = "test mail"
data['fromUserEmail'] = "<sender id>"
data['toUserEmail'] = "<my mail id>"
data['emailContent'] = "test mail"

cookies = "<some headers>"
location = "USA"
content_type = "application/json"
apikey = '<some api key>'
auth = '"Basic <auth key>'

headers = {'X-COM-LOCATION': location, 'Content-Type': content_type, 'apikey': apikey, 'Authorization': auth,
           'Cookie': cookies}
endpoint = "<host> + <path>"

req = requests.Request("POST",url=endpoint,headers=headers,data=data)
ses = requests.Session()
pr = ses.prepare_request(req)
settings = ses.merge_environment_settings(pr.url, {}, None, "/home/airflow/.local/lib/python3.10/site-packages/certifi/cacert.pem",None)
pr.body = 'No, I want exactly this as the body.'
resp = ses.send(pr,**settings)

错误栈如下：

>>> pr.body = 'No, I want exactly this as the body.'
>>> resp = ses.send(pr,**settings)
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/home/airflow/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<host>', port=443): Max retries exceeded with url: <path> (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/airflow/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='<host>', port=443): Max retries exceeded with url: <path> (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))
>>>
>>> print(resp.status_code)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'resp' is not defined

已尝试多种常规方法但问题仍未解决，寻求帮助。

1. 修复证书链缺失问题 #

错误核心是本地信任库缺少API服务器证书的根/中间证书：

• 从API服务方获取完整的证书链文件（.pem格式）
• 将证书添加到Airflow环境的信任库：
  • 对certifi库：把证书追加到/home/airflow/.local/lib/python3.10/site-packages/certifi/cacert.pem末尾
  • 对系统级信任库：将证书放到/etc/ssl/certs/目录，执行update-ca-certificates生效

2. 在Requests中指定自定义证书 #

无需修改全局信任库，直接在请求中指定证书路径：

# 替换为你的证书文件路径
custom_cert = "/path/to/your/api_certificate.pem"
resp = ses.send(pr, verify=custom_cert, **settings)

注意：send方法的verify参数会覆盖merge_environment_settings中的证书配置，直接指定更可靠。

3. Airflow HttpOperator的针对性配置 #

方法A：指定证书路径 #

from airflow.providers.http.operators.http import HttpOperator

http_op = HttpOperator(
    task_id="call_api",
    method="POST",
    endpoint="<endpoint path>",
    http_conn_id="your_api_conn",
    headers=headers,
    data=data,
    verify="/path/to/your/api_certificate.pem"
)

方法B：临时跳过验证（仅测试环境） #

http_op = HttpOperator(
    task_id="call_api",
    method="POST",
    endpoint="<endpoint path>",
    http_conn_id="your_api_conn",
    headers=headers,
    data=data,
    verify=False
)

也可在Airflow连接的Extra字段配置：{"verify": false}或{"verify": "/path/to/cert.pem"}

4. 修正测试代码中的低级错误 #

• endpoint = "<host> + <path>"是字符串拼接错误，应改为endpoint = "<host>" + "<path>"或直接写完整URL
• auth = '"Basic <auth key>'存在引号嵌套错误，正确格式为auth = "Basic <auth key>"

内容的提问来源于stack exchange，提问作者Ayush Goyal