使用requests库遇Max retries exceeded错误，求超时跳过URL方案

使用requests库的get方法检测URL可用性时，多数URL运行正常，但部分URL耗时许久后抛出如下错误：

requests.exceptions.ConnectionError: HTTPConnectionPool(host='macromedia.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001E8ACACF040>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

希望URL超时时自动跳过该链接，但尝试使用continue、pass语句无效，相关代码如下：

import time

import requests
import bs4
from Base_Class import *
import threading

class Checking_Valid_URL:

    def __init__(self):
        self.https = 0
        self.http = 0
        #database connection
        #print(urls)
        self.url_protocols = ['http://', 'https://']
        #database connection

    def Checking_for_http_https_content_status(self):
        for url in self.urls:
            for url_protocol in self.url_protocols:
                try:
                    time.sleep(2)
                    full_https_url = url_protocol + url[0]
                    res = requests.get(full_https_url, timeout=60)
                    soup = bs4.BeautifulSoup(res.text, 'html.parser')
                    elems = soup.select('body')
                    try:
                        if elems:
                            print(f'body found in {full_https_url}')
                            try:
                                if res.status_code == 200:
                                    #database connection
                                    if full_https_url.startswith('https'):
                                        print('https:: ' + full_https_url + ' ' + str(res.status_code))
                                        try:
                                            #database connection
                                            self.https += 1
                                            time.sleep(5)
                                        except:
                                            continue

                                    elif full_https_url.startswith('http'):

                                        print('https:: ' + full_https_url + ' ' + str(res.status_code))
                                        try:
                                            #database connection
                                            self.http += 1
                                            time.sleep(5)
                                        except:
                                            continue
                            except:
                                continue

                        else:
                            print(f"No body in {full_https_url}")
                            continue
                    except:
                        print(f"No body in {full_https_url}")
                        continue

                except requests.exceptions.Timeout:
                    print(f"Timeout on {full_https_url}, skipping")
                    continue
                    
check = Checking_Valid_URL()
check.Checking_for_http_https_content_status()

注：Base_Class包含数据库创建和selenium驱动，#database connection处为数据库相关代码，已移除，功能正常。

解决方案 #

问题根源 #

当前代码仅捕获了requests.exceptions.Timeout异常，但实际抛出的是requests.exceptions.ConnectionError（属于连接失败类异常，并非单纯的请求超时），因此该异常未被捕获，导致continue语句无法执行，程序直接终止。

修改方案 #

1. 扩展异常捕获范围：同时捕获Timeout和ConnectionError，或者直接捕获requests库的顶层异常requests.exceptions.RequestException（覆盖所有请求相关异常）
2. 优化代码结构：减少嵌套try-except层级，提升代码可读性和维护性

修改后的代码示例 #

import time

import requests
import bs4
from Base_Class import *
import threading

class Checking_Valid_URL:

    def __init__(self):
        self.https = 0
        self.http = 0
        #database connection
        #print(urls)
        self.url_protocols = ['http://', 'https://']
        #database connection

    def Checking_for_http_https_content_status(self):
        for url in self.urls:
            for url_protocol in self.url_protocols:
                full_https_url = url_protocol + url[0]
                try:
                    time.sleep(2)
                    res = requests.get(full_https_url, timeout=60)
                    res.raise_for_status()  # 主动触发HTTP状态码异常（如404、500等）
                    
                    soup = bs4.BeautifulSoup(res.text, 'html.parser')
                    elems = soup.select('body')
                    
                    if not elems:
                        print(f"No body in {full_https_url}")
                        continue
                    
                    print(f'body found in {full_https_url}')
                    if res.status_code == 200:
                        #database connection
                        if full_https_url.startswith('https'):
                            print(f'https:: {full_https_url} {res.status_code}')
                            #database connection
                            self.https += 1
                        elif full_https_url.startswith('http'):
                            print(f'http:: {full_https_url} {res.status_code}')  # 修正原代码打印错误
                            #database connection
                            self.http += 1
                        time.sleep(5)

                # 捕获连接超时类异常
                except (requests.exceptions.Timeout, requests.exceptions.ConnectionError):
                    print(f"Connection/Timeout error on {full_https_url}, skipping")
                    continue
                # 捕获其他请求相关异常
                except requests.exceptions.RequestException as e:
                    print(f"Request failed for {full_https_url}: {str(e)}, skipping")
                    continue
                # 捕获意外异常
                except Exception as e:
                    print(f"Unexpected error for {full_https_url}: {str(e)}, skipping")
                    continue
                    
check = Checking_Valid_URL()
check.Checking_for_http_https_content_status()

关键修改点 #

• 新增捕获requests.exceptions.ConnectionError，确保连接失败类异常能被处理并跳过
• 使用res.raise_for_status()主动触发HTTP错误状态码的异常，避免遗漏4xx/5xx类错误
• 简化嵌套try-except结构，将异常捕获统一放在最外层，逻辑更清晰
• 修正原代码中打印http链接时误写为"https::"的问题

内容的提问来源于stack exchange，提问作者stack overflow