从Amazon S3下载图片时文件名追加随机字符串的问题修复

问题详情 #

尝试从Amazon S3存储桶下载图片时，发现文件名末尾被追加了一串随机字母数字组合（如示例中的.ed7c7cDb），同时触发FileNotFoundError错误。以下是相关代码及报错栈：

代码片段 #

import re
import time
import requests
import os, os.path
from urllib.error import HTTPError
import pandas as pd
import sqlalchemy as sa
from sqlalchemy import (
    create_engine,
    MetaData,
    Table,
    Column,
    Integer,
    String,
    Boolean,
    Date,
)
import boto3
import random
import botocore
from datetime import datetime
from urllib.error import HTTPError
import mysql.connector as connection

engine = create_engine(
    "mysql+pymysql://"
    + user
    + ":"
    + passw
    + "@"
    + host
    + ":"
    + str(port)
    + "/"
    + database,
    echo=False,
)

try:
    mydb = connection.connect(
        host="",
        port=int(3306),
        database="db",
        user="",
        passwd="",
        use_pure=True,
    )
    query = "select * from rentalPropertyDetails;"
    df = pd.read_sql(query, mydb)
    mydb.close()  # close the connection
except Exception as e:
    mydb.close()
    print(str(e))

client = boto3.client("s3")
paginator = client.get_paginator("list_objects_v2")
result = paginator.paginate(Bucket="zillowhousephotos")

s3 = boto3.client(
    "s3",
    region_name="us-east-2",
    aws_access_key_id="",
    aws_secret_access_key="",
)

zpids = list(set(df["zpid"].tolist()))

keyStrings = []
picNums = [random.randint(10, 50) for _ in range(len(zpids))]

for page in result:
    if "Contents" in page:
        for item in page["Contents"]:
            keyStrings.append(item["Key"])
for zpid, picNum in zip(zpids, picNums):
    keyString = f"{zpid}/{picNum}.jpg"
    if keyString in keyStrings:
        zpidPicNum = keyString.split(".")[0].replace("/", "_")
        s3.download_file(
            "zillowhousephotos",
            keyString,
            f"/Users/erinoefelein/Desktop/HomeFlip/notFrontOfHouse/{zpidPicNum}"+"jpg",
        )

报错信息 #

Traceback (most recent call last):
  File "/Users/erinoefelein/Desktop/HomeFlip/read_images_to_aws_rds.py", line 88, in <module>
    s3.download_file(
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/boto3/s3/inject.py", line 190, in download_file
    return transfer.download_file(
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/boto3/s3/transfer.py", line 326, in download_file
    future.result()
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/download.py", line 642, in _main
    fileobj.seek(offset)
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/utils.py", line 378, in seek
    self._open_if_needed()
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/utils.py", line 361, in _open_if_needed
    self._fileobj = self._open_function(self._filename, self._mode)
  File "/Users/erinoefelein/.local/share/virtualenvs/HomeFlip-71a8tVCO/lib/python3.10/site-packages/s3transfer/utils.py", line 272, in open
    return open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/erinoefelein/Desktop/HomeFlip/notFrontOfHouse/29417540_17.jpg.ed7c7cDb'

异常文件名示例：29417540_17.jpg.ed7c7cDb

原因分析 #

• 目标目录不存在：boto3的s3.download_file方法在下载时会先创建带随机后缀的临时文件，待下载完成后再重命名为目标文件名。如果指定的目标目录（notFrontOfHouse）不存在，临时文件无法创建，就会触发FileNotFoundError，此时报错信息里显示的是未完成的临时文件名。
• 文件名拼接错误：代码中拼接文件名时写的是f"...{zpidPicNum}"+"jpg"，这会导致生成的文件名是29417540_17jpg（缺少.），虽然这不是触发当前报错的直接原因，但会导致最终保存的文件名不符合预期。

解决方法 #

1. 提前创建目标目录
   在下载前检查并创建目标目录，使用os.makedirs并设置exist_ok=True，避免因目录不存在导致的错误：

   target_dir = "/Users/erinoefelein/Desktop/HomeFlip/notFrontOfHouse"
   os.makedirs(target_dir, exist_ok=True)
   

2. 修正文件名拼接错误
   将文件名拼接代码中的+"jpg"改为+".jpg"，或者直接在f-string中完成拼接，确保文件名格式正确：

   # 修正后的文件名拼接方式
   local_filename = f"/Users/erinoefelein/Desktop/HomeFlip/notFrontOfHouse/{zpidPicNum}.jpg"
   s3.download_file("zillowhousephotos", keyString, local_filename)
   

3. 完整修改后的关键代码片段

   target_dir = "/Users/erinoefelein/Desktop/HomeFlip/notFrontOfHouse"
   os.makedirs(target_dir, exist_ok=True)
   
   for zpid, picNum in zip(zpids, picNums):
       keyString = f"{zpid}/{picNum}.jpg"
       if keyString in keyStrings:
           zpidPicNum = keyString.split(".")[0].replace("/", "_")
           local_filename = f"{target_dir}/{zpidPicNum}.jpg"
           s3.download_file("zillowhousephotos", keyString, local_filename)
   

内容的提问来源于stack exchange，提问作者Erin