循环处理SFTP文件时运行19条后遇paramiko SSHException错误求助

问题根源 #

短时间内频繁创建SSH/SFTP连接，触发了服务器的连接次数限制，导致连接被重置。原代码每处理一条记录就新建一次连接，19次后达到服务器阈值，抛出SSHException。

解决方案 #

1. 核心优化：复用SSH连接 #

只创建一次SSH连接，在循环中复用该连接完成所有下载操作，大幅减少连接请求次数，避免触发服务器限制。

2. 修复其他潜在问题 #

• 日志写入从覆盖模式(w)改为追加模式(a)，保留所有失败记录
• 自动创建本地存储目录，避免因目录不存在报错
• 细化异常捕获，便于排查具体错误
• 确保连接最终关闭，避免资源泄漏

修改后的完整代码 #

import paramiko
import os
import time 

lsit1 = ["CBCGG1900089HM", "CBCGG1900090HM", "CBCGG1900091HM", "CBCGG1900092HM", "CBCGG1900093HM", "CBCGG1900094HM", "CBCGG1900095HM", "CBCGG1900096HM", "CBCGG1900097HM", "CBCGG1900098HM", "CBCGG1900099HM", "CBCGG1900100", "CBCGG1900101HM", "CBCGG1900102HM", "CBCGG1900103HM", "CBCGG1900104HM", "CBCGG1900105HM", "CBCGG1900106HM", "CBCGG1900108HM", "CBCGG1900109HM", "CBCGG1900112HM", "CBCGG1900116HM", "CBCGG1900120HM", "CBCGG1900124HM", "CBCGG1900128HM", "CBCGG1900130HM", "CBCGG1900131HM", "CBCGG1900132HM", "CBCGG1900136HM", "CBCGG1900142HM", "CBCGG1900149HM", "CBCGG1900151HM", "CBCGG1900153", "CBCGG1900161", "CBDGG1900001", "BCYG1800038", "BCYG1800041", "BCYG1800045", "BCYG1800047", "BCYG1800049"]
col_id = []

def sftp_download(ftp, item):
    try:
        # 创建本地存储目录
        local_dir = f'/home/hvr/sftp_python/Optima660_Beijing {item}'
        if not os.path.exists(local_dir):
            os.makedirs(local_dir)
        # 执行下载
        remote_path = f'/CT_Logs/Optima660_Beijing/{item}/image_analysis.csv'
        local_path = f'{local_dir}/image_analysis.csv'
        ftp.get(remote_path, local_path)
        print(f"{item} 下载完成")
    except Exception as e:
        error_msg = f"{item} 下载失败: {str(e)}"
        print(error_msg)
        col_id.append(f"Optima660_Beijing/{item}")
        # 追加写入日志
        with open("/home/hvr/sftp_python/logs/sftp_logs.txt", 'a') as f:
            f.write(f"{error_msg}\n")

# 主逻辑：建立一次连接，批量下载
try:
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(hostname='abc.com', timeout=6000, username='abc', password='xxbb')
    ftp = ssh.open_sftp()
    print("SFTP连接已建立")

    for idx, item in enumerate(lsit1):
        sftp_download(ftp, item)
        # 每15次下载后休眠，降低服务器压力
        if idx % 15 == 0 and idx != 0:
            time.sleep(15)

finally:
    # 确保连接最终关闭
    if 'ftp' in locals():
        ftp.close()
    if 'ssh' in locals():
        ssh.close()
    # 写入最终失败列表
    if col_id:
        with open("/home/hvr/sftp_python/logs/sftp_logs.txt", 'a') as f:
            f.write("\n下载失败的记录列表:\n")
            f.write("\n".join(col_id))

额外优化建议 #

• 如果服务器限制单连接的操作次数，可以每20-30次下载后重新建立一次连接
• 增加重试机制，对偶尔的网络波动或服务器繁忙进行自动重试
• 根据服务器实际情况调整休眠时间，避免操作过于密集

内容的提问来源于stack exchange，提问作者Thentu matcha