SQS触发Lambda的批处理功能未按预期工作

Hey there, let's break down exactly what's going on here and why your Lambda isn't behaving like you expected.

The key issue here is how AWS Lambda scales with SQS triggers when there's a backlog of messages.

You set a batch size of 250 and a batch window of 65 seconds, but Lambda doesn't wait 65 seconds between batches when there's a large number of messages in the queue (like your 32,000). Instead, Lambda automatically scales out concurrent execution instances to process the backlog faster. Each instance grabs a batch of 250 messages, and if there are enough messages, Lambda will spin up as many instances as allowed by your account's Lambda concurrency limits (or until the queue's visibility timeout prevents more messages from being fetched).

In your case, Lambda spun up 60 concurrent instances (60 * 250 = 15,000) all at once. Since your third-party API's token bucket has a maximum capacity of 15,000 tokens, all those instances drained the bucket immediately. After that, the token bucket only refills at 250 tokens per minute—way too slow to keep up with any additional batches, hence the failure to process the remaining messages.

Your expectation of "one batch every 65 seconds" only holds when there's a small number of messages (not enough to trigger multiple concurrent instances). With a large backlog, Lambda prioritizes throughput over strict batch timing.

Here are a few ways to fix this and align Lambda's behavior with your API's rate limits:

• Limit Lambda Concurrency to 1
  The simplest fix is to set your Lambda function's reserved concurrency to 1. This ensures only one instance runs at a time, so it'll process one batch of 250 messages every ~65 seconds. Since 250 messages per 65 seconds is roughly 230 calls per minute—just under your API's 250 per minute limit—this will perfectly match the token bucket's refill rate, and you won't hit the max token capacity upfront.

  To set this: Go to your Lambda function's configuration → Concurrency → Reserved concurrency, and set it to 1.

• Adjust Batch Window and Batch Size
  If you want to stick closer to exactly 250 calls per minute, tweak the batch window to 60 seconds instead of 65. With concurrency set to 1, this means Lambda will process one batch of 250 messages every minute, aligning perfectly with the API's token refill rate.

• Implement Cross-Instance Rate Limiting (Advanced)
  If you need higher throughput but still want to respect the API's limits, you can build a shared rate limiter across all Lambda instances. For example, use DynamoDB to track the number of tokens used in the last minute, and have each instance check before making API calls. This is more complex but gives you flexibility if you need to scale beyond 250 messages per minute later (though your API's limits would still cap you there).

Your Lambda didn't follow the "one batch every 65 seconds" pattern because it scaled out to handle the large message backlog. By limiting concurrency to 1, you force it to process batches sequentially, which matches your API's token bucket constraints.

内容的提问来源于stack exchange，提问作者nats