升级Python3.10后使用boto3上传下载文件遇RuntimeError求助
Python3.10下boto3/awswrangler大文件上传下载触发RuntimeError问题
在Python3.8环境中,boto3.client.download_file()和boto3.client.upload_file()函数运行正常,但升级至Python3.10后,出现以下错误:
RuntimeError: cannot schedule new futures after interpreter shutdown
已测试boto3多个版本(1.17、1.18、1.26及最新版),均存在该问题。
报错堆栈信息
Traceback (most recent call last): File "/opt/.venv/lib/python3.10/site-packages/awswrangler/s3/_fs.py", line 585, in open_s3_object yield s3obj File "/opt/.venv/lib/python3.10/site-packages/awswrangler/s3/_upload.py", line 74, in upload s3_f.write(local_f.read()) # type: ignore[arg-type] File "/opt/.venv/lib/python3.10/site-packages/awswrangler/s3/_fs.py", line 550, in write self.flush() File "/opt/.venv/lib/python3.10/site-packages/awswrangler/s3/_fs.py", line 427, in flush self._upload_proxy.upload( File "/opt/.venv/lib/python3.10/site-packages/awswrangler/s3/_fs.py", line 133, in upload future = self._exec.submit( File "/usr/lib/python3.10/concurrent/futures/thread.py", line 169, in submit raise RuntimeError('cannot schedule new futures after ' RuntimeError: cannot schedule new futures after interpreter shutdown
已尝试的解决方案(均不满足需求)
- 禁用线程:可解决错误,但大文件上传下载效率显著下降,希望保留线程以维持性能
- 调用
thread.join()等待线程结束:无效,仍触发相同错误 - 使用awswrangler库:小文件可正常运行,大文件仍会触发该错误
寻求能保留线程功能、同时解决该RuntimeError的有效方案。
内容的提问来源于stack exchange,提问作者djGowda
相关产品推荐
相关产品推荐

