使用boto3上传CSV至S3时触发FileNotFoundError报错排查

操作流程 #

尝试使用Boto3将CSV文件上传至名为ebayunited的S3存储桶，执行步骤如下：

• 从测试数据源接口拉取产品JSON格式数据
• 将JSON数据转换为CSV格式并保存至脚本所在同级目录
• 调用Boto3 SDK执行文件上传操作

原始实现代码 #

import requests
import pandas as pd
import boto3 as bt

from API_keys import access_key, secret_access_key
client = bt.client(
    's3', aws_access_key_id=access_key,
          aws_secret_access_key=secret_access_key
)

r = requests.get("https://dummyjson.com/products")
json_products = pd.DataFrame(r.json()["products"])
file_s3 = json_products.to_csv("Orders.csv", index=False)

bucket = "ebayunited"
path = "Lysi Team/ebayapi/"+str(file_s3)

client.upload_file(str(file_s3), bucket, path)

报错信息 #

执行代码后抛出如下异常：

Traceback (most recent call last):
  File "c:\Users\PC\Desktop\S3 Import\s3_import.py", line 18, in <module>
    client.upload_file(str(file_s3), bucket, path)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\boto3\s3\inject.py", line 143, in upload_file
    return transfer.upload_file(
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\boto3\s3\transfer.py", line 288, in upload_file
    future.result()
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\s3transfer\futures.py", line 103, in result
    return self._coordinator.result()
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\s3transfer\futures.py", line 266, in result
    raise self._exception
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\s3transfer\tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\s3transfer\upload.py", line 585, in _submit
    upload_input_manager.provide_transfer_size(transfer_future)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\s3transfer\upload.py", line 244, in provide_transfer_size
    self._osutil.get_file_size(transfer_future.meta.call_args.fileobj)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\s3transfer\utils.py", line 247, in get_file_size
    return os.path.getsize(filename)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'None'

核心报错为系统找不到指定文件'None'。

问题根因 #

错误来自对pandas to_csv()方法返回值的误用：

• 当to_csv()传入具体文件路径参数时，方法会直接将CSV内容写入指定路径的本地文件，返回值固定为None，不会返回生成的文件路径。
• 代码中用file_s3变量接收了这个None返回值，后续拼接S3存储路径、调用upload_file()时，传入的本地文件参数实际是str(None)也就是字符串"None"，Boto3会尝试读取本地名为None的文件执行上传，自然触发文件不存在的报错。

修复方案 #

单独定义本地文件名变量，不要使用to_csv()的返回值作为文件路径，修正后代码如下：

import requests
import pandas as pd
import boto3 as bt
import os

from API_keys import access_key, secret_access_key
client = bt.client(
    's3', 
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_access_key
)

r = requests.get("https://dummyjson.com/products")
json_products = pd.DataFrame(r.json()["products"])

# 单独定义本地CSV文件路径，用绝对路径避免工作目录不对找不到文件
local_file_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "Orders.csv")
# 写入本地CSV，不需要接收返回值
json_products.to_csv(local_file_path, index=False)

bucket = "ebayunited"
# 拼接S3端的存储路径
s3_save_path = "Lysi Team/ebayapi/Orders.csv"

# 传入正确的本地文件路径执行上传
client.upload_file(local_file_path, bucket, s3_save_path)

补充说明：代码里用os模块拼接了脚本所在目录的绝对路径作为CSV文件存储位置，可以避免因运行脚本时的工作目录和脚本所在目录不一致，导致的本地文件找不到问题。

内容的提问来源于stack exchange，提问作者Idriss