Python Selenium遇ERR_TIMED_OUT超时错误时自动刷新页面的实现方案（无需逐行添加try-except）

Hey there! Dealing with rotating proxies can be a real headache when random failures crash your script—especially that annoying ERR_TIMED_OUT error. Let’s break down some practical solutions to handle this globally, so you don’t have to wrap every single Selenium command in try-except blocks.

1. Custom WebDriver Class with Built-in Retries & Refresh #

The cleanest way to avoid repetitive error handling is to create a custom WebDriver that overrides common methods (like get(), find_element()) to automatically handle timeouts. This way, every call to these methods will include retry logic out of the box.

Here’s an example for ChromeDriver:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException, WebDriverException

class ProxyRetryDriver(webdriver.Chrome):
    def __init__(self, max_retries=3, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.max_retries = max_retries
        # Set reasonable timeouts to trigger errors faster
        self.set_page_load_timeout(15)
        self.set_script_timeout(15)

    def get(self, url):
        retries = 0
        while retries < self.max_retries:
            try:
                super().get(url)
                return  # Success, exit the loop
            except (TimeoutException, WebDriverException) as e:
                if "ERR_TIMED_OUT" in str(e):
                    print(f"Proxy timed out! Refreshing and retrying ({retries+1}/{self.max_retries})")
                    self.refresh()
                    retries += 1
                else:
                    raise  # Re-raise non-timeout exceptions

        # If all retries fail, raise the final exception
        raise TimeoutException(f"Failed to load {url} after {self.max_retries} attempts")

    # Extend to other methods like element lookup
    def find_element(self, by, value):
        retries = 0
        while retries < self.max_retries:
            try:
                return super().find_element(by, value)
            except TimeoutException as e:
                print(f"Element lookup timed out, refreshing page...")
                self.refresh()
                retries += 1
        raise TimeoutException(f"Could not find element {value} after {self.max_retries} retries")

How to use it: #

Instead of initializing webdriver.Chrome(), use your custom class:

driver = ProxyRetryDriver(max_retries=3)
driver.get("https://your-target-url.com")

This keeps your main script clean and ensures all critical operations have built-in recovery logic.

2. Global Exception Hook (Careful Use!) #

If you want to catch unhandled exceptions globally, you can use Python’s sys.excepthook. Note that this is trickier because you’ll need to track what your script was doing when the error occurred to retry properly.

Example:

import sys
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.remote.webdriver import WebDriver

# Store your driver instance globally (or pass it via a wrapper)
driver: WebDriver = None

def global_error_handler(exctype, value, traceback):
    # Check if it's a proxy-related timeout error
    if exctype is TimeoutException and "ERR_TIMED_OUT" in str(value):
        print("Caught proxy timeout! Refreshing page...")
        if driver:
            driver.refresh()
            # Optional: Add logic to re-run the last failed operation
            # (This requires tracking script state, which adds complexity)
    else:
        # Let Python handle other exceptions normally
        sys.__excepthook__(exctype, value, traceback)

sys.excepthook = global_error_handler

Caveat: #

This works for catching errors, but automatically retrying the failed action is difficult unless you track the state of your script. It’s better for logging and basic recovery than full retry automation.

3. Use a Retry Library (e.g., Tenacity) #

For specific operations (like page loads), you can use a retry library like tenacity to wrap your calls without modifying the Driver class.

First install it: pip install tenacity

Then use it in your script:

from tenacity import retry, stop_after_attempt, retry_if_exception_type
from selenium.common.exceptions import TimeoutException

@retry(
    stop=stop_after_attempt(3),
    retry=retry_if_exception_type(TimeoutException),
    before_sleep=lambda retry_state: print(f"Retrying... Attempt {retry_state.attempt_number}/3")
)
def load_with_retry(driver, url):
    driver.get(url)

# Usage
load_with_retry(driver, "https://your-target-url.com")

This is great for one-off operations where you don’t want to modify the entire Driver class.

Bonus: Prevent Timeouts Before They Happen #

Beyond handling errors after they occur, you can reduce their frequency:

• Pre-validate proxies: Before assigning a proxy to your Driver, send a quick request (e.g., to http://icanhazip.com) to check if it’s working. Skip invalid proxies immediately.
• Proxy pool health checks: If you manage your own proxy pool, regularly test proxies and remove dead ones to avoid using them in the first place.
• Tweak Selenium timeouts: Set shorter page load/script timeouts so errors trigger faster, reducing wait time before recovery.

内容的提问来源于stack exchange，提问作者DevInprogress