上传Tempfile至S3预签名URL时出现Faraday管道破裂错误求助

在Ruby on Rails应用中，使用Faraday Ruby gem将30MB及以上的MP4视频文件（以Tempfile形式存储）通过PUT请求流式传输至Amazon S3预签名URL时，频繁触发Faraday::ConnectionFailed: Broken pipe错误。该错误非必现，操作成功率约50%，失败后重试大多能成功。

环境版本：

• Ruby 2.3.1
• Rails 5.0.0
• Faraday 0.17.3（使用net/http适配器，切换至httpclient后出现类似HTTPClient::KeepAliveDisconnected: Broken pipe错误，偏好net/http）

相关代码：

headers = { content_type: 'video/mp4', content_length: tempfile.size.to_s }
conn = Faraday.new(url: presigned_url)
resp = conn.put(presigned_url_path, tempfile, headers)

曾尝试在请求中设置req.options.timeout = 180，但无改善。完整堆栈跟踪：

WARN: Faraday::ConnectionFailed: Broken pipe
WARN: /usr/local/lib/ruby/2.3.0/openssl/buffering.rb:322:in `syswrite'
/usr/local/lib/ruby/2.3.0/openssl/buffering.rb:322:in `do_write'
/usr/local/lib/ruby/2.3.0/openssl/buffering.rb:340:in `write'
/usr/local/lib/ruby/2.3.0/net/http/generic_request.rb:206:in `copy_stream'
/usr/local/lib/ruby/2.3.0/net/http/generic_request.rb:206:in `send_request_with_body_stream'
/usr/local/lib/ruby/2.3.0/net/http/generic_request.rb:123:in `exec'
/usr/local/bundle/gems/aws-sdk-core-3.15.0/lib/seahorse/client/net_http/patches.rb:28:in `block in new_transport_request'
/usr/local/bundle/gems/aws-sdk-core-3.15.0/lib/seahorse/client/net_http/patches.rb:27:in `catch'
/usr/local/bundle/gems/aws-sdk-core-3.15.0/lib/seahorse/client/net_http/patches.rb:27:in `new_transport_request'
/usr/local/lib/ruby/2.3.0/net/http.rb:1407:in `request'
/usr/local/lib/ruby/2.3.0/net/http.rb:1400:in `block in request'
/usr/local/lib/ruby/2.3.0/net/http.rb:853:in `start'
/usr/local/lib/ruby/2.3.0/net/http.rb:1398:in `request'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/adapter/net_http.rb:87:in `perform_request'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/adapter/net_http.rb:43:in `block in call'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/adapter/net_http.rb:92:in `with_net_http_connection'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/adapter/net_http.rb:38:in `call'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/request/url_encoded.rb:15:in `call'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/rack_builder.rb:143:in `build_response'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/connection.rb:387:in `run_request'
/usr/local/bundle/gems/faraday-0.17.3/lib/faraday/connection.rb:175:in `put'

• 连接主动中断：Broken pipe本质是数据传输过程中，连接被S3或中间网络设备（防火墙、代理）主动关闭，常见触发场景：
  • 超时配置不全：仅设置请求总超时，未覆盖大文件上传所需的写入超时，导致传输中途被断开。
  • TCP缓冲区处理问题：Ruby net/http流式传输大文件时，缓冲区写入不及时，引发远端判定连接闲置并关闭。
  • 预签名URL有效期不足：上传过程中URL过期，S3直接断开连接。
  • 网络波动：长传输过程中，中间节点临时中断连接。

1. 完善Faraday超时配置 #

针对net/http适配器，需同时配置连接、读取、写入三类超时，覆盖大文件传输的全流程：

conn = Faraday.new(url: presigned_url) do |faraday|
  faraday.adapter :net_http do |http|
    http.open_timeout = 30    # 连接建立超时
    http.read_timeout = 300   # 响应读取超时
    http.write_timeout = 300  # 数据写入超时，适配大文件
  end
end

2. 采用S3分块上传（推荐） #

S3原生支持分块上传，将大文件拆分为多个小块传输，某块失败仅需重传对应部分，大幅提升可靠性：

require 'aws-sdk-s3'

s3 = Aws::S3::Client.new(region: '你的区域')
file = tempfile

# 初始化分块上传
upload = s3.create_multipart_upload(bucket: '你的存储桶', key: '文件存储路径')
parts = []

# 分块大小设为5MB（可调整为10-100MB）
chunk_size = 5 * 1024 * 1024
file_size = file.size
total_parts = (file_size.to_f / chunk_size).ceil

(1..total_parts).each do |part_number|
  start = (part_number - 1) * chunk_size
  finish = [start + chunk_size - 1, file_size - 1].min
  part_size = finish - start + 1

  file.seek(start)
  chunk = file.read(part_size)

  # 上传单个分块
  resp = s3.upload_part(
    bucket: '你的存储桶',
    key: '文件存储路径',
    part_number: part_number,
    upload_id: upload.upload_id,
    body: chunk
  )

  parts << { part_number: part_number, etag: resp.etag }
end

# 完成分块上传合并
s3.complete_multipart_upload(
  bucket: '你的存储桶',
  key: '文件存储路径',
  upload_id: upload.upload_id,
  multipart_upload: { parts: parts }
)

3. 添加指数退避重试机制 #

针对非必现的网络错误，用faraday-retry中间件实现自动重试，降低失败概率：

# 先在Gemfile添加依赖
gem 'faraday-retry'

# 配置重试规则
conn = Faraday.new(url: presigned_url) do |faraday|
  faraday.request :retry,
    max: 3,                  # 最大重试次数
    interval: 1,             # 初始重试间隔（秒）
    backoff_factor: 2,       # 指数退避因子
    exceptions: [Faraday::ConnectionFailed] # 指定需要重试的异常
  faraday.adapter :net_http do |http|
    http.open_timeout = 30
    http.read_timeout = 300
    http.write_timeout = 300
  end
end

4. 延长预签名URL有效期 #

确保URL有效期足够覆盖最长上传时间，比如设置为1小时：

# 若使用aws-sdk生成预签名URL
s3 = Aws::S3::Resource.new
obj = s3.bucket('你的存储桶').object('文件存储路径')
presigned_url = obj.presigned_url(:put, expires_in: 3600) # 有效期3600秒

5. 调整TCP缓冲区参数（进阶） #

通过修改TCP发送/接收缓冲区大小，减少写入阻塞概率：

conn = Faraday.new(url: presigned_url) do |faraday|
  faraday.adapter :net_http do |http|
    http.open_timeout = 30
    http.read_timeout = 300
    http.write_timeout = 300
    http.socket_options = {
      sndbuf: 1024 * 1024, # 发送缓冲区设为1MB
      rcvbuf: 1024 * 1024  # 接收缓冲区设为1MB
    }
  end
end

优先选择S3分块上传方案，这是针对大文件传输的原生可靠方案；其次通过完善超时配置、添加重试机制覆盖网络波动场景；同时确保预签名URL有效期充足，可有效降低错误发生概率。

内容的提问来源于stack exchange，提问作者Matt S.