本地环境AWS SDK v3上传S3随机请求超时问题求助

问题描述 #

将AWS SDK从v2迁移至v3后，本地环境上传文件到S3时随机出现以下错误：

Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

部署后的站点无此问题，使用PutObjectCommand处理小文件、Upload分片上传两种场景均会随机触发该错误，相关代码如下：

const saveS3 = (fileName, path, target, awsBucket, contentDirectory) => {
  return new Promise(async (resolve, reject) => {
    fs.stat(path, (err, stats) => {
      if (err) {
        return reject({
          success: false,
          message: "S3 error reading file from local source",
          fileUrl: null,
          fileName: null,
          storageType: target,
        });
      }

      const fileSize = stats.size;

      fs.readFile(path, async (err, data) => {
        if (err) {
          return reject({
            success: false,
            message: "S3 error reading file from local source",
            fileUrl: null,
            fileName: null,
            storageType: target,
          });
        }
        const params = {
          Bucket: awsBucket,
          Key: `${contentDirectory}/${fileName}`,
          Body: data,
        };

        if (fileSize <= 5 * 1024 * 1024) {
          console.log("small File")
          const command = new PutObjectCommand(params);
          console.log("command",command)
          s3Client.send(command)
            .then(() => {
              console.log("inside then")
              AWSSignedURL(params.Key, awsBucket)
                .then((url) => {
                  return resolve({
                    success: true,
                    message: "File uploaded to S3",
                    fileUrl: url.url,
                    fileName: `${contentDirectory}/${fileName}`,
                    storageType: target,
                  });
                })
                .catch((err) => {
                  console.log(err);
                  return reject({
                    success: false,
                    message: "Error obtaining url",
                    fileUrl: null,
                    fileName: null,
                    storageType: target,
                  });
                });
            })
            .catch((err) => {
              return reject({
                success: false,
                message: ("S3 error uploading file to remote source :", err),
                fileUrl: null,
                fileName: null,
                storageType: target,
              });
            });
        } else {
          console.log("large File")
          const upload = new Upload({
            client: s3Client,
            params: params,
          });
          upload.done().then((data) => {
            console.log("dataaaa", data)
            AWSSignedURL(data.Key, awsBucket)
              .then((url) => {
                return resolve({
                  success: true,
                  message: "File uploaded to S3",
                  fileUrl: url.url,
                  fileName: `${contentDirectory}/${fileName}`,
                  storageType: target,
                });
              })
              .catch((err) => {
                console.log(err);
                return reject({
                  success: false,
                  message: "Error obtaining url",
                  fileUrl: null,
                  fileName: null,
                  storageType: target,
                });
              });
          }).catch((err) => {
            return reject({
              success: false,
              message: ("S3 error uploading file to remote source :", err),
              fileUrl: null,
              fileName: null,
              storageType: target,
            });
          });
        }
      });
    });
  });
}; 

解决方法 #

1. 调整S3客户端超时与连接配置 #

AWS SDK v3默认超时设置可能不适应本地网络，初始化客户端时手动配置更长超时时间并开启长连接：

import { S3Client } from "@aws-sdk/client-s3";
import https from "https";

const s3Client = new S3Client({
  region: "你的AWS区域",
  credentials: {
    accessKeyId: "你的Access Key ID",
    secretAccessKey: "你的Secret Access Key"
  },
  requestTimeout: 30000, // 请求超时30秒
  connectionTimeout: 10000, // 连接超时10秒
  maxAttempts: 3, // 自动重试次数
  retryMode: "standard", // 标准重试模式
  httpOptions: {
    timeout: 30000,
    agent: new https.Agent({
      keepAlive: true, // 启用长连接
      maxSockets: 50 // 最大并发连接数
    })
  }
});

2. 改用流式上传替代内存读取 #

当前代码将整个文件读入内存后上传，大文件易导致传输超时。改用文件流直接上传，降低内存占用同时提升传输稳定性：

// 替换fs.readFile为流式读取
const fileStream = fs.createReadStream(path);
fileStream.on("error", (err) => {
  return reject({
    success: false,
    message: "S3 error reading file stream",
    fileUrl: null,
    fileName: null,
    storageType: target,
  });
});

const params = {
  Bucket: awsBucket,
  Key: `${contentDirectory}/${fileName}`,
  Body: fileStream // 直接传递流
};

// 小文件上传
const command = new PutObjectCommand(params);
// 大文件上传
const upload = new Upload({
  client: s3Client,
  params: params
});

3. 针对超时错误添加手动重试 #

在上传逻辑中捕获超时错误，进行手动重试：

// 封装带重试的上传函数
const uploadWithRetry = async (commandOrUpload, isUpload = false) => {
  const maxRetries = 3;
  for (let i = 0; i < maxRetries; i++) {
    try {
      return isUpload ? await commandOrUpload.done() : await s3Client.send(commandOrUpload);
    } catch (err) {
      if ((err.name === "TimeoutError" || err.message.includes("timeout")) && i < maxRetries - 1) {
        console.log(`上传超时，重试第${i+1}次`);
        continue;
      }
      throw err;
    }
  }
};

// 使用示例：小文件上传
uploadWithRetry(command)
  .then(/* 后续逻辑 */)
  .catch(/* 错误处理 */);

// 大文件上传
uploadWithRetry(upload, true)
  .then(/* 后续逻辑 */)
  .catch(/* 错误处理 */);

4. 优化本地网络环境 #

• 检查本地防火墙、代理工具是否拦截了S3的出站连接
• 切换至有线网络或更稳定的WiFi环境
• 关闭后台占用带宽的程序，减少网络拥堵

5. 更新AWS SDK至最新稳定版 #

旧版本SDK可能存在连接相关bug，执行以下命令更新依赖：

npm update @aws-sdk/client-s3 @aws-sdk/lib-storage

内容的提问来源于stack exchange，提问作者SmartLearn