使用多进程调用boto3 S3 upload_fileobj触发SSLError问题求助

问题 #

运行环境：AWS Lambda（Python 3.9，boto3 1.20.32）
代码功能：从S3桶的tar文件中按字节范围提取单个文件，通过多进程上传至另一S3桶，加速大量文件的传输。

核心代码：

s3_client = boto3.client(service_name="s3")
s3_bucket = "bucket"
s3_other_bucket = "other_bucket"

def multiprocess_s3upload(tar_index: dict):

    def _upload(filename, bytes_range):
        src_key = ...

        # 按字节范围获取tar中的单个文件
        s3_obj = s3_client.get_object(
            Bucket=s3_bucket,
            Key=src_key,
            Range=f"bytes={bytes_range}"
        )

        # 上传文件（报错位置）
        s3_client.upload_fileobj(
            s3_obj["Body"],
            s3_other_bucket,
            filename
        )

    def _wait(procs):
        for p in procs:
            p.join()
    
    processes = []
    proc_limit = 256  # 限制并发数避免"打开过多文件"错误
    for filename, bytes_range in tar_index.items():
        proc = Process(
            target=_upload,
            args=(filename, bytes_range)
        )
        proc.start()
        processes.append(proc)
        
        if len(processes) == proc_limit:
            _wait(processes)
            processes = []

    _wait(processes)

报错现象：随机抛出SSLError（仅最后一个子进程报错），报错信息：

Process Process-2:
Traceback (most recent call last):
File "/var/runtime/urllib3/response.py", line 441, in _error_catcher
  yield
File "/var/runtime/urllib3/response.py", line 522, in read
  data = self._fp.read(amt) if not fp_closed else b""
File "/var/lang/lib/python3.9/http/client.py", line 463, in read
  n = self.readinto(b)
File "/var/lang/lib/python3.9/http/client.py", line 507, in readinto
  n = self.fp.readinto(b)
File "/var/lang/lib/python3.9/socket.py", line 704, in readinto
  return self._sock.recv_into(b)
File "/var/lang/lib/python3.9/ssl.py", line 1242, in recv_into
  return self.read(nbytes, buffer)
File "/var/lang/lib/python3.9/ssl.py", line 1100, in read
  return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2633)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lang/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
  self._target(*self._args, **self._kwargs)
File "/var/task/main.py", line 144, in _upload
  s3_client.upload_fileobj(
File "/var/runtime/boto3/s3/inject.py", line 540, in upload_fileobj
  return future.result()
File "/var/runtime/s3transfer/futures.py", line 103, in result
  return self._coordinator.result()
File "/var/runtime/s3transfer/futures.py", line 266, in result
  raise self._exception
File "/var/runtime/s3transfer/tasks.py", line 269, in _main
  self._submit(transfer_future=transfer_future, **kwargs)
File "/var/runtime/s3transfer/upload.py", line 588, in _submit
  if not upload_input_manager.requires_multipart_upload(
File "/var/runtime/s3transfer/upload.py", line 404, in requires_multipart_upload
  self._initial_data = self._read(fileobj, threshold, False)
File "/var/runtime/s3transfer/upload.py", line 463, in _read
  return fileobj.read(amount)
File "/var/runtime/botocore/response.py", line 82, in read
  chunk = self._raw_stream.read(amt)
File "/var/runtime/urllib3/response.py", line 544, in read
  raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
  self.gen.throw(typ, value, traceback)
File "/var/runtime/urllib3/response.py", line 452, in _error_catcher
  raise SSLError(e)

urllib3.exceptions.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2633)

已知背景：排查过旧问题（非线程安全HTTP库）但无效，需要稳定解决方案。

解决方案 #

1. 每个子进程创建独立的S3客户端 #

boto3客户端并非多进程安全，跨进程共享会导致连接池、SSL上下文冲突。修改_upload函数，在子进程内部初始化S3客户端：

def _upload(filename, bytes_range):
    # 子进程内单独创建S3客户端
    s3_client = boto3.client(service_name="s3")
    src_key = ...

    s3_obj = s3_client.get_object(
        Bucket=s3_bucket,
        Key=src_key,
        Range=f"bytes={bytes_range}"
    )

    s3_client.upload_fileobj(
        s3_obj["Body"],
        s3_other_bucket,
        filename
    )

2. 改用多线程替代多进程 #

Lambda环境中多进程开销更大，且boto3客户端是线程安全的（每个线程使用独立客户端或线程安全连接池即可）。用ThreadPoolExecutor替换Process：

from concurrent.futures import ThreadPoolExecutor

def multiprocess_s3upload(tar_index: dict):
    def _upload(filename, bytes_range):
        s3_client = boto3.client(service_name="s3")
        src_key = ...

        s3_obj = s3_client.get_object(
            Bucket=s3_bucket,
            Key=src_key,
            Range=f"bytes={bytes_range}"
        )

        s3_client.upload_fileobj(
            s3_obj["Body"],
            s3_other_bucket,
            filename
        )
    
    # Lambda环境建议线程数不超过100，避免资源耗尽
    with ThreadPoolExecutor(max_workers=100) as executor:
        executor.map(_upload, tar_index.keys(), tar_index.values())

3. 降低并发数 #

当前设置的256并发数过高，Lambda的文件句柄、网络连接资源有限，容易触发SSL异常。将proc_limit调整为64或更低，测试出适合当前Lambda配置的最优值。

4. 添加重试机制 #

针对SSLError这类偶发网络异常，添加指数退避重试逻辑：

from botocore.exceptions import ClientError
import time
import ssl

def _upload(filename, bytes_range):
    s3_client = boto3.client(service_name="s3")
    src_key = ...
    max_retries = 3

    for attempt in range(max_retries):
        try:
            s3_obj = s3_client.get_object(
                Bucket=s3_bucket,
                Key=src_key,
                Range=f"bytes={bytes_range}"
            )

            s3_client.upload_fileobj(
                s3_obj["Body"],
                s3_other_bucket,
                filename
            )
            break
        except (ClientError, ssl.SSLError) as e:
            if attempt == max_retries -1:
                raise
            # 指数退避重试
            time.sleep(2 ** attempt)

5. 升级boto3版本 #

boto3 1.20.32属于较旧版本，后续版本修复了多个多进程/线程下的连接问题。如果Lambda环境允许，升级到最新稳定版（如1.34.x）。

内容的提问来源于stack exchange，提问作者Chiamin