Swift：URLSession实现S3大文件无内存加载预签名POST上传

摘要 #

我通过boto3的generate_presigned_post生成了S3预签名POST URL和字段，需要上传数百GB级别的大文件到S3。以下curl命令已成功完成上传：

curl -X POST \
  # 复制的字段
  -F key='filename.mp4' -F AWSAccessKeyId='AKIA...EWS3' -F policy='eyJle...dfQ==' -F signature='RcU9...XLw=' \
  # 要上传的文件
  -F file=@filename.mp4 \
  # 目标URL
  https://some-bucket.s3.amazonaws.com/

我尝试用原生URLSession复刻上述curl调用，不想引入重型依赖，但自行实现时遇到了阻碍。

详细说明与代码 #

我尝试了两种方法：使用URLSession.upload(for: fromFile:)（内部基于UploadTask）和URLSession.data(for:)（后者需要手动将文件数据设置到httpBody）。

我编写了一个辅助方法，将键值对和边界转换为表单字段：

private func convertFormField(named name: String, value: String, using boundary: String) -> String
{
    var fieldString = "--\(boundary)\r\n"
    fieldString += "Content-Disposition: form-data; name=\"\(name)\"\r\n"
    fieldString += "Content-Type: text/plain; charset=ISO-8859-1\r\n"
    fieldString += "Content-Transfer-Encoding: 8bit\r\n"
    fieldString += "\r\n"
    fieldString += "\(value)\r\n"

    return fieldString
}

该方法用于处理key、AWSAccessKeyId、policy、signature字段，我原本期望URLSession.upload(for:fromFile:)能为指定URL的文件设置合适的HTTP请求体数据并分块，且兼容multipart/form-data POST请求，但实际调用后客户端超时。

上传方法代码：

func upload(fileUrl:URL, toS3:S3Payload) async throws -> Bool
{
    print("Uploading", fileUrl, toS3)
    
    guard let url = URL(string: toS3.url) else {
        return false
    }
    
    let attributes = try FileManager.default.attributesOfItem(atPath: fileUrl.path() )
    
    let fileSize = attributes[.size] as? NSNumber
    
    var request = URLRequest(url:url)
    let boundary = "Boundary-\(UUID().uuidString)"

    var httpBody = Data()

    for (k,v) in toS3.fields
    {
        let string = convertFormField(named: k, value: v, using: boundary)
        print("Field", string)
        let data = string.data(using: .utf8)
        guard let data = data else { continue }
        httpBody.append(data)
    }
            
    let totalHTTPBodyBytes = httpBody.count
    
    request.httpBody = httpBody
            
    request.httpMethod = "POST"
    request.timeoutInterval = 30
    request.setValue("*/*", forHTTPHeaderField: "Accept")
    request.setValue(String( totalHTTPBodyBytes + ( fileSize?.intValue ?? 0 ) ), forHTTPHeaderField: "Content-Length")
    request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
    request.setValue("100-continue", forHTTPHeaderField: "Expect")

    print("All Header Fields", request.allHTTPHeaderFields)

    // 使用独立的URLSession，避免传递内部API的认证令牌
    let (data, response ) = try await self.s3URLSession.upload(for: request, fromFile: fileUrl )
      
    guard let httpResponse = response as? HTTPURLResponse,
          httpResponse.statusCode == 204 else {
            return false
    }

    
    return true
}

错误信息 #

调用后出现客户端超时错误：

2023-06-19 15:14:08.615747-0400 Ozu[41759:3883308] Task <3ADB06DD-6F17-4F2A-AABB-233A13A387B3>.<1> finished with error [-1001] Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x6000035952f0 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalUploadTask <3ADB06DD-6F17-4F2A-AABB-233A13A387B3>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalUploadTask <3ADB06DD-6F17-4F2A-AABB-233A13A387B3>.<1>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://xxx.s3.amazonaws.com/, NSErrorFailingURLKey=https://xxx.s3.amazonaws.com/, _kCFStreamErrorDomainKey=4}
Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x6000035952f0 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalUploadTask <3ADB06DD-6F17-4F2A-AABB-233A13A387B3>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalUploadTask <3ADB06DD-6F17-4F2A-AABB-233A13A387B3>.<1>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://xxx.s3.amazonaws.com/, NSErrorFailingURLKey=https://xxx.s3.amazonaws.com/, _kCFStreamErrorDomainKey=4}
2023-06-19 15:14:08.633225-0400 Ozu[41759:3883606] [tcp] tcp_input [C5.1.1.1:3] flags=[R.] seq=4101766915, ack=1634103508, win=251 state=CLOSED rcv_nxt=4101766884, snd_una=1634103477

疑问 #

• 超时是否因为服务器等待我未发送的多部分块？
• self.s3URLSession.upload(for: request, fromFile: fileUrl )是否能正确将文件添加到请求体的其余部分？
• 如果该方法不支持多部分上传，如何在不将整个文件加载到内存的情况下为httpBody提供大文件数据？

任何建议都将不胜感激！

更新 #

我已通过URLSession.data(for:)实现上传，但该方法需要将整个文件加载到内存，并不理想。真的无法使用URLSession.upload(for:fromFile:)实现无内存加载的上传吗？

内容的提问来源于stack exchange，提问作者vade