为何Node.js 18.x Lambda函数无法复制S3对象？

我创建了基于Node.js 18.x运行时的Lambda函数，能正常从S3桶获取对象列表，但复制操作无法执行，尽管函数返回状态显示“Successful”。复制命令前的console.log能输出，但命令后的console.log没有输出。

代码实现 #

import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"

const client = new S3Client({})

export const handler = async (event) => {
  const sourceBucket = 'in-bucket'
  const destBucket = 'complete-bucket'
  const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')
  const commandList = new ListObjectsV2Command({Bucket: sourceBucket})

  try {
    let isTruncated = true

    while (isTruncated) {
      const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
      Contents.forEach(async (c) => {
        const fn = c.Key
        const options = {
          CopySource: encodeURI(`${sourceBucket}/${fn}`),
          Bucket: destBucket,
          Key: encodeURI(`${fn}${dateSuffix}`)
        }
        console.log(`moving ${fn}`)
        const commandCopy = new CopyObjectCommand(options)
        let response = await client.send(commandCopy)
        console.log(`response: ${response}`)
      })
      isTruncated = IsTruncated
      commandList.input.ContinuationToken = NextContinuationToken
    }
    console.log('Finished')

  } catch (err) {
    console.error(err)
  }
}

执行结果 #

Test Event Name
(unsaved) test event

Response
null

Function Logs
START RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Version: $LATEST
2023-07-27T17:50:28.279Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file01.csv
2023-07-27T17:50:28.281Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file02.csv
2023-07-27T17:50:28.337Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file03.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file04.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file05.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file06.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file07.csv
2023-07-27T17:50:28.339Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file08.csv
2023-07-27T17:50:28.339Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file09.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file10.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file11.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file12.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file13.csv
2023-07-27T17:50:28.341Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file14.csv
2023-07-27T17:50:28.341Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file15.csv
2023-07-27T17:50:28.397Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file16.csv
2023-07-27T17:50:28.397Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    Finished
END RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
REPORT RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2  Duration: 1178.43 ms    Billed Duration: 1179 ms    Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 621.73 ms

Request ID
dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2

我参考了官方示例代码，第一个命令能返回所有预期键值，但复制操作未按预期执行。

问题根源 #

1. forEach不等待异步任务：代码中使用Contents.forEach(async (c) => { ... })遍历对象列表，但forEach不会等待内部异步函数执行完成。Lambda函数会在while循环结束后直接执行console.log('Finished')并退出，此时所有复制操作的异步任务还未完成，导致复制失败，后续日志也无法输出。
2. 不必要的编码：CopySource和Key不需要手动用encodeURI编码，AWS SDK会自动处理特殊字符，手动编码可能导致路径解析错误。
3. 命令状态污染：直接修改commandList.input.ContinuationToken可能导致循环中命令状态混乱，建议每次循环创建新的ListObjectsV2Command实例。

修正后的代码 #

import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"

const client = new S3Client({})

export const handler = async (event) => {
  const sourceBucket = 'in-bucket'
  const destBucket = 'complete-bucket'
  const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')

  try {
    let continuationToken = null
    let isTruncated = true

    while (isTruncated) {
      // 每次循环创建新的List命令，避免状态污染
      const commandList = new ListObjectsV2Command({
        Bucket: sourceBucket,
        ContinuationToken: continuationToken
      })
      const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
      
      // 使用for...of等待每个复制操作完成
      for (const c of Contents) {
        const fn = c.Key
        const options = {
          CopySource: `${sourceBucket}/${fn}`,
          Bucket: destBucket,
          Key: `${fn}${dateSuffix}`
        }
        console.log(`moving ${fn}`)
        const commandCopy = new CopyObjectCommand(options)
        const response = await client.send(commandCopy)
        console.log(`response: ${JSON.stringify(response)}`) // 用JSON.stringify打印完整响应对象
      }
      
      isTruncated = IsTruncated
      continuationToken = NextContinuationToken
    }
    console.log('Finished')

  } catch (err) {
    console.error(err)
    throw err // 抛出错误让Lambda标记为失败，便于排查问题
  }
}

可选优化：并行复制 #

如果需要提升复制效率，可使用Promise.all并行处理所有复制任务（注意控制并发数，避免触发S3速率限制）：

// 替换代码中的for...of部分
const copyPromises = Contents.map(async (c) => {
  const fn = c.Key
  const options = {
    CopySource: `${sourceBucket}/${fn}`,
    Bucket: destBucket,
    Key: `${fn}${dateSuffix}`
  }
  console.log(`moving ${fn}`)
  const commandCopy = new CopyObjectCommand(options)
  const response = await client.send(commandCopy)
  console.log(`response: ${JSON.stringify(response)}`)
})
await Promise.all(copyPromises)

内容的提问来源于stack exchange，提问作者Ed Trembicki-Guy