Requests重试策略失效问题及超时重试需求实现咨询

Hey there, let's figure out why your retry mechanism isn't kicking in. The core issue here is that your try-except blocks in get() and post() are cutting off the retry process before it even gets started. Let's break down the problem and fix it step by step.

What's Going Wrong? #

• Early Exception Capture: Your current code catches requests.exceptions.RequestException immediately after the first failed request. This includes timeout errors, so you're throwing your MyException right away before the HTTPAdapter has a chance to trigger any retries.
• POST Retry Limitation: By default, Retry doesn't retry POST requests (to prevent duplicate submissions). If you need retries for POST, you have to explicitly allow it.
• Incomplete Retry Configuration: Your Retry setup doesn't specify how retries should handle wait intervals, and we can make it clearer which errors trigger retries.

Fixed Code #

from requests import Session
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util import Retry
from my_exception import MyException
import requests

# Configure the retry strategy properly
retry_strategy = Retry(
    total=2,  # Total number of retries (initial request + 2 retries = 3 total attempts)
    connect=2,
    read=2,
    status_forcelist=[500, 502, 503, 504],
    allowed_methods=["GET", "POST"],  # Allow retries for both GET and POST (adjust as needed)
    backoff_factor=1,  # Wait 1s, 2s, 4s... between retries (optional but recommended)
)

s = Session()
adapter = HTTPAdapter(max_retries=retry_strategy)
s.mount("http://", adapter)
s.mount("https://", adapter)

def get(url):
    try:
        r = s.get(url, timeout=(1, 60))
        r.raise_for_status()
        return r
    except requests.exceptions.RequestException as e:
        # Now this only runs after ALL retries have failed
        if isinstance(e, (requests.exceptions.ConnectTimeout, requests.exceptions.ReadTimeout)):
            raise MyException(500, "Requests timeout after all retries.")
        # Handle non-timeout errors (e.g., 4xx/5xx after retries)
        status_code = e.response.status_code if hasattr(e, 'response') else 500
        raise MyException(status_code, str(e))

def post(url, data):
    try:
        r = s.post(url, data=data, timeout=(1, 60))
        r.raise_for_status()
        return r
    except requests.exceptions.RequestException as e:
        if isinstance(e, (requests.exceptions.ConnectTimeout, requests.exceptions.ReadTimeout)):
            raise MyException(500, "Requests timeout after all retries.")
        status_code = e.response.status_code if hasattr(e, 'response') else 500
        raise MyException(status_code, str(e))

Key Changes Explained #

1. Let Retries Run First: We removed the early exception handling that was stopping retries. Now the try-except only triggers after the HTTPAdapter has exhausted all retry attempts (initial request + 2 retries = 3 total attempts).
2. Explicitly Allow POST Retries: Added allowed_methods=["GET", "POST"] to the Retry config. If you don't need POST retries, you can omit this (it defaults to ["GET", "HEAD"]).
3. Better Timeout Detection: Instead of checking the exception message string, we use isinstance() to directly check for timeout exception types—this is more reliable than string matching.
4. Safe Status Code Handling: For non-timeout errors, we check if the exception has a response attribute before accessing status_code to avoid AttributeErrors.
5. Backoff Factor: The backoff_factor=1 adds a delay between retries (1s, 2s, 4s...) to avoid overwhelming the server. You can adjust this or remove it if you don't need delays.

内容的提问来源于stack exchange，提问作者Jael