Python批量检测网站状态异常：仅apple.com可用，其余报SSL错误

问题背景 #

使用以下Python代码检测CSV文件中的网站状态时，仅apple.com能正常返回状态，其余网站均抛出HTTPSConnectionPool、SSLCertVerificationError等SSL相关错误；但将CSV中所有URL替换为apple.com时，无论顺序都能正常检测。

检测代码 #

import csv
import requests
from http import HTTPStatus
from fake_useragent import UserAgent


def get_websites(csv_path: str) -> list[str]:
    """Loads websites from a csv file"""

    websites: list[str] = []
    with open(csv_path, 'r') as file:
        reader = csv.reader(file)
        for row in reader:
            if 'https://' not in row[1]:
                websites.append(f'https://{row[1]}')
            else:
                websites.append(row[1])

    return websites


def get_user_agent() -> str:
    """Returns a user agent that can be used with requests"""

    ua = UserAgent()
    return ua.chrome


def get_status_description(status_code: int) -> str:
    """Uses the status code to return a readable description"""

    for value in HTTPStatus:
        if value == status_code:
            description: str = f'({value} {value.name}) {value.description}'
            return description

    return '(???) Unknown status code...'


def check_website(website: str, user_agent: str):
    """Gets that status code for a website and prints the result"""
    try:
        response = requests.get(website, headers={'User-Agent': user_agent})
        code: int = response.status_code
        print(website, get_status_description(code))
    except Exception as e:
        print("Exception is: ", e)


def main():
    sites: list[str] = get_websites('websites.csv')
    user_agent: str = get_user_agent()

    # Check websites
    for i, site in enumerate(sites):
        check_website(site, user_agent)


if __name__ == '__main__':
    main()

CSV文件内容 #

1,"apple.com"
2,"facebook.com"
3,"lulumall.com"
4,"newgrounds.com"
5,"linkedin.com"
6,"python-guide.com"
7,"fastmail.com"
8,"soundcloud.com"
9,"whatsapp.com/hello"
10,"udemy.com"

运行输出 #

Exception is:  HTTPSConnectionPool(host='linkedin.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='facebook.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='lulumall.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] 
EOF occurred in violation of protocol (_ssl.c:1006)')))
https://apple.com (200 OK) Request fulfilled, document follows
Exception is:  HTTPSConnectionPool(host='linkedin.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='python-guide.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='fastmail.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='soundcloud.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='whatsapp.com', port=443): Max retries exceeded with url: /hello (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Exception is:  HTTPSConnectionPool(host='udemy.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

解决方案 #

1. 临时跳过SSL验证（仅用于测试，生产环境不推荐） #

在requests.get调用中添加verify=False参数，同时关闭不安全请求警告：

# 在文件顶部添加
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# 修改check_website函数中的请求代码
response = requests.get(website, headers={'User-Agent': user_agent}, verify=False)

2. 更新本地CA证书 #

requests依赖系统默认的CA证书库，如果证书缺失或过期会导致验证失败。可以：

• 手动指定CA证书路径：在requests.get中添加verify="/path/to/cacert.pem"
• 更新系统CA证书（Windows可通过系统更新补丁，macOS使用brew install openssl更新，Linux安装ca-certificates包）

3. 更新相关依赖库 #

旧版本的requests或urllib3可能存在证书处理bug，执行以下命令更新：

pip install --upgrade requests urllib3 fake-useragent

4. 检查网络环境 #

如果处于代理、VPN或企业防火墙后，这些设备可能会替换SSL证书导致验证失败。此时需要配置requests使用代理，并确保代理证书被信任：

# 示例：配置HTTP代理
proxies = {
    'http': 'http://your-proxy:port',
    'https': 'http://your-proxy:port',
}
response = requests.get(website, headers={'User-Agent': user_agent}, proxies=proxies)

内容的提问来源于stack exchange，提问作者Vaishnavi Vaishu