Python捕获Exception后程序仍崩溃？设备重连异常排查求助

我正在开发一款自动化程序，用于在特定节点重启设备。为实现设备重启后的重连，我希望在等待一段时间后持续尝试连接直至成功。本以为通过捕获所有Exception，在连接失败时休眠60秒后递归调用连接函数即可，但程序仍因连接错误崩溃，出现requests.exceptions.ConnectionError、urllib3.exceptions.MaxRetryError等多种异常，后续补充的完整回溯显示最终抛出ciscoisesdk.exceptions.ciscoisesdkException。

我的连接函数 #

def establish_session(device):
    try:
        sess = IdentityServicesEngineAPI(username=config.username,
                                    password=config.password,
                                    base_url='https://' + device,
                                    verify=False)
    except Exception as e:
        print("connection refused. node rebooting. Trying again in 60 seconds")
        time.sleep(60)
        return establish_session(device) # Nested here...

    return sess

完整异常回溯 #

Traceback (most recent call last):
  File "/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/python3.10/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x104e2b670>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='device1', port=443): Max retries exceeded with url: /api/v1/task (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x104e2b670>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python3.10/site-packages/ciscoisesdk/restsession.py", line 497, in request
    response = self._req_session.request(method, abs_url, **kwargs)
  File "/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='device1', port=443): Max retries exceeded with url: /api/v1/task (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x104e2b670>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/python3.10/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x104e29930>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='device1', port=443): Max retries exceeded with url: /api/v1/task (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x104e29930>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.10/site-packages/ciscoisesdk/restsession.py", line 504, in request
    response = self._req_session.request(method, abs_url,
  File "/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='device1', port=443): Max retries exceeded with url: /api/v1/task (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x104e29930>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Pre_Post_Checks_Automation/test_api_calls.py", line 5, in <module>
    hold = api.tasks.get_task_status().response
  File "python3.10/site-packages/ciscoisesdk/api/v3_1_patch_1/tasks.py", line 123, in get_task_status
    _api_response = self._session.get(endpoint_full_url, params=_params)
  File "/python3.10/site-packages/ciscoisesdk/restsession.py", line 609, in get
    response = self.request('GET', url, erc, 0, params=params, **kwargs)
  File "python3.10/site-packages/ciscoisesdk/restsession.py", line 507, in request
    raise ciscoisesdkException('Socket error {}'.format(e))
ciscoisesdk.exceptions.ciscoisesdkException: Socket error HTTPSConnectionPool(host='device1', port=443): Max retries exceeded with url: /api/v1/task (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x104e29930>: Failed to establish a new connection: [Errno 61] Connection refused'))

排查与修复建议 #

1. 问题根源 #

你的代码在调用IdentityServicesEngineAPI时成功创建了会话，但后续使用会话调用get_task_status()时才抛出异常——而这个异常并没有被establish_session函数的try-except捕获，因为异常发生在会话创建之后的API调用阶段，不是会话初始化阶段。另外，递归调用连接函数存在栈溢出风险，长时间重试可能触发RecursionError。

2. 修复方案1：给API调用添加重试逻辑 #

封装通用的重试装饰器，把重试逻辑覆盖到所有可能抛出连接异常的API调用：

import time
from functools import wraps
import requests
import urllib3
from ciscoisesdk.exceptions import ciscoisesdkException

def retry_on_connection_errors(delay=60):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            while True:
                try:
                    return func(*args, **kwargs)
                except (requests.exceptions.ConnectionError, 
                        urllib3.exceptions.MaxRetryError, 
                        ciscoisesdkException) as e:
                    print(f"Connection error: {e}. Retrying in {delay}s...")
                    time.sleep(delay)
        return wrapper
    return decorator

# 使用装饰器包装API调用
@retry_on_connection_errors()
def get_task_status(api):
    return api.tasks.get_task_status().response

3. 修复方案2：验证会话有效性后再返回 #

在establish_session中创建会话后，立即执行一个轻量API调用（比如获取版本信息）验证连接可用性，同时用循环代替递归避免栈溢出：

import time
from ciscoisesdk import IdentityServicesEngineAPI

def establish_session(device):
    while True:
        try:
            sess = IdentityServicesEngineAPI(username=config.username,
                                        password=config.password,
                                        base_url='https://' + device,
                                        verify=False)
            # 执行轻量API调用验证会话
            sess.version.get_version()
            print("Connection established successfully")
            return sess
        except Exception as e:
            print(f"Connection failed: {e}. Retrying in 60s...")
            time.sleep(60)

4. 额外优化点 #

• 精准捕获异常：避免捕获所有Exception，明确列出需要处理的连接类异常，防止意外忽略配置错误、权限问题等其他错误。
• 指数退避策略：可以修改重试间隔，比如第一次60秒，第二次120秒，第三次240秒，避免频繁请求给设备造成压力。
• 日志替代print：使用Python的logging模块记录错误信息，方便后续调试和问题排查。

内容的提问来源于stack exchange，提问作者Jack McGowan