Python脚本调用requests.get()遇Max retries exceeded with URL错误求助

Hey there! Let's break down why you're seeing that Max retries exceeded error and get your script working smoothly.

What's Causing the Error? #

The error pops up because the URLs your script generates (like http://www.googla.com) don't point to real, resolvable domains. When requests tries to connect to these invalid URLs, it automatically retries the request a few times by default. Once all those retries fail, it throws the "Max retries exceeded" exception, which stops your script mid-execution.

Modified Script with Error Handling & Retry Control #

Here's an updated version of your script that handles these failures gracefully, lets you control retries, and adds timeouts to prevent hanging requests:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Set up a session with custom retry rules
session = requests.Session()
# Configure retry behavior: total=0 disables retries entirely
retry_strategy = Retry(
    total=0,
    backoff_factor=0.1,
    status_forcelist=[429, 500, 502, 503, 504]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)

base_prefix = "http://www.googl"
base_suffix = ".com"
start_char = 'a'

for x in range(0, 8):
    current_char = chr(ord(start_char) + x)
    target_url = f"{base_prefix}{current_char}{base_suffix}"
    try:
        # Add a timeout to avoid waiting indefinitely for invalid URLs
        response = session.get(target_url, timeout=3)
        print(f"{target_url} | Status Code: {response.status_code} | Elapsed Time: {response.elapsed.total_seconds()}s")
    except requests.exceptions.RequestException as e:
        # Catch all request-related errors and keep the script running
        print(f"{target_url} | Request Failed: {str(e)}")

Key Improvements Explained #

• Custom Retry Strategy: By setting total=0, we disable requests default retry behavior entirely. If you want some retries for temporary issues, you can bump this number (e.g., total=2).
• Timeout Setting: The timeout=3 ensures each request gives up after 3 seconds instead of hanging indefinitely for unresolvable domains.
• Exception Handling: The try-except block catches any request errors (DNS failures, timeouts, connection issues) and prints the error instead of crashing the script.
• Cleaner String Formatting: Using f-strings makes the URL construction easier to read and maintain.

Extra Tips #

If you ever want to test valid URLs with specific response codes, you can use dedicated test endpoints, but since you mentioned invalid URLs are okay, your original URL generation logic works perfectly here.

内容的提问来源于stack exchange，提问作者Wisam Ahmed