使用AWS SDK Firehose v3与NodeJS 16.19.1持续出现OOM错误求助

在使用AWS SDK Firehose v3版本搭配NodeJS 16.19.1时，应用持续触发内存不足（OOM）错误。应用会直接向AWS Firehose发送大量PutRecordCommand请求，且未触发限流。现咨询：这是AWS SDK Firehose v3的已知问题，还是实现存在根本性错误？

相关代码 #

const DeliveryStreamName = 'REDACTED'

const command = new PutRecordCommand({
  DeliveryStreamName,
  Record: { Data: Buffer.from(REDACTED) },
})

const client = new FirehoseClient({
  region: 'us-east-1',
  endpoint: REDACTED,
  credentials: REDACTED
})

await client.send(command)

错误信息 #

已设置NODE_OPTIONS=--max-old-space-size=8192参数，但仍出现OOM错误：

<--- Last few GCs --->
[252:0x5d16cf0]   189663 ms: Mark-sweep 8111.2 (8235.3) -> 8102.4 (8238.3) MB, 325.8 / 0.1 ms  (average mu = 0.404, current mu = 0.054) allocation failure scavenge might not succeed
[252:0x5d16cf0]   190004 ms: Mark-sweep 8114.2 (8239.7) -> 8105.8 (8241.9) MB, 324.4 / 0.1 ms  (average mu = 0.260, current mu = 0.050) allocation failure GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb08e80 node::Abort() [node]
 2: 0xa1b70e  [node]
 3: 0xce1890 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xce1c37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xe992a5  [node]
 6: 0xea8f6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xeabc6e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xe6cee2 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
 9: 0xe654f4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
10: 0xe67200 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
11: 0x10f85e8 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [node]
12: 0xced189 v8::String::Utf8Length(v8::Isolate*) const [node]
13: 0xade517  [node]
14: 0xd3ddfe  [node]
15: 0xd3f21f v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
16: 0x15d9cf9  [node]
Aborted (core dumped)

问题分析与解决建议 #

1. 实现层面的根本性错误

• 重复创建FirehoseClient：若每次发送请求都实例化新的FirehoseClient，会导致大量未回收的连接、客户端实例占用内存。需全局复用一个客户端实例，而非每次请求都新建。
• 未控制并发量：大量PutRecordCommand请求并发发送，会导致内存中堆积大量未完成的Promise、请求对象、Buffer数据，NodeJS事件循环无法及时回收资源。建议用限流工具（如p-limit）控制并发数，或改用PutRecordBatchCommand批量发送，减少请求次数。
• 单条请求Buffer堆积：若REDACTED是大体积数据，大量单个请求的Buffer会持续占用内存。批量请求能显著降低Buffer总占用。

2. SDK层面的已知问题

AWS SDK v3部分旧版本曾存在高频请求场景下的内存泄漏问题，建议检查当前@aws-sdk/client-firehose版本，升级到最新稳定版尝试解决。

3. 排查步骤

• 将FirehoseClient初始化代码放在全局作用域，复用实例；
• 替换为PutRecordBatchCommand，批量发送最多500条/4MB的记录；
• 用限流工具将并发请求数控制在合理范围（如100以内，根据实际测试调整）；
• 使用node --inspect启动应用，结合Chrome DevTools内存分析工具定位泄漏对象。

内容的提问来源于stack exchange，提问作者steadweb