AWS S3批量下载在第40个文件后失败的技术求助

问题背景 #

此前正常运行的AWS S3批量下载代码，近期出现每次下载第40个文件后失败的问题，且未修改过相关配置。客户端浏览器收到catch块返回的错误提示：

"Unable to download the zip file. Please retry on a high speed internet connection or contact support, if issue persists!"

目前无法支持用户下载超过40个文件。

已排查场景 #

• 更换不同图片测试，问题依旧
• 更换不同S3文件夹测试，问题依旧
• 更换不同用户账号测试，问题依旧
• 更换不同S3存储桶测试，问题依旧
• 在staging和production服务器分别测试，问题均存在
• 服务器存储、内存及文件权限无异常，预签名URL可正常单独访问

疑问点 #

1. 是否存在文件未完成下载就开始填充$imageZip数组的情况？
2. 若存在上述情况，为何每次都精准在第40个文件失败？文件大小不同，下载时长应存在差异。

相关脱敏代码 #

下载Zip核心方法 #

public function downloadZip(Request $request) { 

    $reqdata = $request->all();
    $ordref = @$reqdata['ordref'];
    $usrnumber = @$reqdata['usrnumber'];

    if (!$ordref || !$usrnumber) {
    //check if parameters passed valid
        echo json_encode($data);
        exit();
    } 

    $this->db = DB::table("orders AS o");
    $details = $this->db->select("o.photos", "o.ref", "o.id", "o.folder_id", "o.price","o.is_download")
            ->where(....)
            ->first();

    if (!$details) {
        return redirect("/");
    }

    //check if json is valid
    $images = $this->processphotos($details->photos);

    // find photo ids from DB        
    ....
    ....
    $dbimages = $this->getimagesbyids($imageids);

    $imageNames = [];
    foreach ($dbimages as $item) {
        $thumbName = $item->imgurl;
        array_push($imageNames, array('image' => $thumbName, 'category' => $item->category, 'id' => $item->id));
    }


    if (!$imageNames) {
    //return No images found error.

    }

    $zipimages = [];
    $public_dir = public_path() . DIRECTORY_SEPARATOR . 'downloads' . DIRECTORY_SEPARATOR;
    $zipimagesarray = array();
    $imagepatharray = explode("/", $imageNames[0]['image']);
    $imagename = end($imagepatharray);

    $ctr = 100;
    foreach ($imageNames as $row) {
        $imgtype = 'jpg';
        $prefix = $usrnumber.'_';
        $imagename = $prefix . $ctr . $imgtype;
        $signedurl = $this->getpresignedurl($row['image']);
        @ File::copy($signedurl, $public_dir . $imagename);

        array_push($zipimagesarray, array('path' => $public_dir, "image" => $imagename));
        $ctr++;
    }

    $imageZip = [];
    $certZip = [];
    foreach ($zipimagesarray as $imagePublic) {
        $path = explode(DIRECTORY_SEPARATOR,$imagePublic['path']);

        $zipImage = $imagePublic['path'] . $imagePublic['image'];
        array_push($imageZip, $zipImage);
    }

    try {
        $zipName = $usrnumber . date('G:i:s') . ".zip";

        Zipper::make(public_path('downloadOrder/' . $zipName))->add($imageZip)->close();

        File::delete($imageZip);
        $file = "/download/" . $zipName;

        $data['result'] = array();
        $data['result'] = url('/') . $file ;
        $data['statusCode'] = 200;
        $data['hasError'] = false;
        $data['message'] = "Images retrieval is successful.";
        return response($data)->header('Content-Type', 'application/json');
    } catch (\Exception $ex) {
        $data['result'] = array();
        $data['statusCode'] = 400;
        $data['hasError'] = true;
        $data['message'] = "Unable to download the zip file. Please retry on a high speed internet connection or contact support, if issue persists!";
        Log::error("Api::downloadZip - orderId: " . $data['Message'] . " " . $ex->getMessage());
        return response($data)->header('Content-Type', 'application/json');
    }
}

预签名URL生成方法 #

function getpresignedurl($key, $s3Client = null) {

$s3_env = [];
$Bucket = "xxxxx";

if (is_null($s3Client)) { 
   $s3_env = array('region' => config('aws.AWS_REGION'),
       'credentials' => ['key' => config('aws.AWS_ACCESS_KEY_ID'), 'secret' => config('aws.AWS_SECRET_ACCESS_KEY')]);
   $s3Client = \App::make('aws')->createClient('s3', $s3_env);
}

//Creating a presigned URL
$cmd = $s3Client->getCommand('GetObject', [
    'Bucket' => $Bucket,
    'Key' => $key
]);

$request = $s3Client->createPresignedRequest($cmd, '+30 minutes');
$signedurl = (string) $request->getUri();
return $signedurl;
}

问题分析与解决建议 #

核心问题定位 #

1. 错误抑制掩盖下载失败：代码中用@ File::copy(...)抑制了所有错误，包括S3文件下载失败的情况。当第40次请求触发限制导致下载失败时，本地文件不存在，但代码仍会把无效路径加入打包数组，最终Zipper打包时抛出异常。
2. 固定40次失败的根源：这是AWS S3请求速率限制或服务器并发连接数限制导致的——每次循环都会创建新的S3客户端，同步发起下载请求，当累计到40次时触发限流，导致下载失败。

解决步骤 #

1. 移除错误抑制，添加下载验证：
   去掉@符号，检查File::copy的返回值，确认文件下载成功后再加入打包队列：

   $success = File::copy($signedurl, $public_dir . $imagename);
   if (!$success) {
       Log::error("Failed to download file from: " . $signedurl);
       throw new \Exception("Failed to download image: " . $row['image']);
   }
   

2. 复用S3客户端：在downloadZip方法初始化时创建一次S3客户端，传入getpresignedurl避免重复创建：

   // 循环前创建S3客户端
   $s3Client = \App::make('aws')->createClient('s3', [
       'region' => config('aws.AWS_REGION'),
       'credentials' => ['key' => config('aws.AWS_ACCESS_KEY_ID'), 'secret' => config('aws.AWS_SECRET_ACCESS_KEY')]
   ]);
   
   // 循环中调用时传入客户端
   $signedurl = $this->getpresignedurl($row['image'], $s3Client);
   

3. 添加请求延迟或并发控制：针对速率限制，可在每次下载后添加短暂延迟：

   // 每次下载后延迟100毫秒
   usleep(100000);
   

4. 检查AWS配额与日志：登录AWS控制台，查看CloudWatch中的S3请求指标，确认是否有429 Too Many Requests错误，同时检查账号的服务配额是否被调整。
5. 优化打包流程：跳过本地下载环节，直接通过S3客户端流式读取文件到Zip，减少存储依赖和下载时间。

内容的提问来源于stack exchange，提问作者Krishna Chebrolu