处理图片下载时的异常URL：UnicodeEncodeError与ssl.CertificateError

Hey there! Let's work through the two issues you're hitting when downloading ~6600 images from URLs in your pickle file. I've dealt with similar problems before, so here's how to fix them:

The root issue here is that URLs with non-English characters need proper encoding before you can send HTTP requests—browsers handle this automatically, but Python doesn't. Here are two targeted fixes depending on where the non-ASCII characters are:

If non-ASCII characters are in the URL path (e.g., /images/中文图片.jpg) #

Use urllib.parse.quote() to encode just the path segment without breaking the scheme or domain:

from urllib.parse import quote, urlparse, urlunparse

def sanitize_url_path(url):
    parsed_url = urlparse(url)
    # Encode only the path part to handle non-ASCII characters
    encoded_path = quote(parsed_url.path)
    # Reconstruct the full URL with the encoded path
    return urlunparse((
        parsed_url.scheme,
        parsed_url.netloc,
        encoded_path,
        parsed_url.params,
        parsed_url.query,
        parsed_url.fragment
    ))

Usage: Pass every URL from your pickle file through this function before attempting to download.

If non-ASCII characters are in the domain (e.g., 中文网站.com) #

You'll need to convert the domain to punycode using the idna module (built into Python 3.4+):

from urllib.parse import urlparse, urlunparse

def sanitize_idna_domain(url):
    parsed_url = urlparse(url)
    # Encode the domain to punycode (IDNA format)
    encoded_domain = parsed_url.netloc.encode('idna').decode('utf-8')
    # Reconstruct the full URL
    return urlunparse((
        parsed_url.scheme,
        encoded_domain,
        parsed_url.path,
        parsed_url.params,
        parsed_url.query,
        parsed_url.fragment
    ))

For URLs with both non-ASCII domains and paths, combine both functions: first sanitize the domain, then the path.

This error happens when the SSL certificate presented by the server doesn't match the hostname you're requesting. Here are your options, ordered by safety:

Option 1: Verify the correct hostname #

Sometimes the issue is as simple as using the wrong domain (e.g., requesting www.example.com when the certificate only covers example.com, or vice versa). Double-check if the URL uses the correct domain prefix—you can test this by visiting the URL in a browser to see if it throws a certificate warning.

Option 2: Use the server's CA certificate (Recommended for production) #

If the server uses a custom or self-signed certificate, you can download the CA certificate file and tell your HTTP library to trust it. For example, with requests:

import requests

# Replace with the path to your downloaded CA certificate file
response = requests.get(url, verify='/path/to/ca-certificate.crt')

Option 3: Skip certificate verification (Only for testing/non-sensitive data) #

If you're just testing and don't care about security (never do this for production code), you can disable verification and suppress the warning:

import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning

# Disable the insecure request warning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Make the request without verifying the certificate
response = requests.get(url, verify=False)

Putting it all together, your workflow would be:

1. Load URLs from the pickle file
2. Sanitize each URL with the appropriate encoding function
3. Attempt to download, handling SSL errors with one of the above methods

内容的提问来源于stack exchange，提问作者Mona Jalal