Hey Patrick, let's break down this frustrating SQS issue you're dealing with—your 4-second delay queue shouldn't be hanging onto messages for over a minute when traffic is light and you've got plenty of Lambda capacity. Here are some targeted areas to investigate:

Potential Causes & Fixes #

1. Check for Improperly Deleted Messages #

The ApproximateAgeOfOldestMessage metric counts from a message's original send time, not when it re-enters the queue after visibility timeout. If your Lambda isn't handling message deletion correctly (even silently), messages will loop back into the queue and their age will keep increasing over retries.

• Dig into your Lambda's CloudWatch logs to look for:
  • Unhandled exceptions that cause the entire batch to be requeued
  • Cases where the Lambda completes successfully but fails to delete messages (though the SQS-Lambda integration usually handles deletion automatically on success)
  • Execution times approaching your Lambda's 20-second timeout—partial failures here can lead to messages being requeued

2. Switch to SQS Long Polling #

By default, Lambda uses short polling for SQS, which might return empty responses even if messages are in the queue. This can lead to delays in message retrieval as Lambda waits for the next poll cycle.

• Enable long polling for your SQS queue by setting WaitTimeSeconds to 20 (the maximum value) in your queue configuration. This ensures SQS waits for messages to arrive before responding to poll requests, reducing empty polls and speeding up message processing.

3. Enable Batch Item Failures for Lambda #

If you're using batch processing (even with batch size 1), a single failed message in a batch will cause the entire batch to be requeued. This creates a loop where the same messages are retried repeatedly, inflating the oldest message age.

• In your Lambda's SQS trigger settings, enable Report Batch Item Failures. This lets your Lambda return only the IDs of failed messages, so SQS will requeue those specific messages while deleting the successful ones. This prevents entire batches from looping unnecessarily.

4. Verify Message Delay Configuration #

Double-check that all messages are actually being sent with a 4-second delay:

• Confirm your queue's Delivery Delay is set to 4 seconds (not a higher value)
• If you're setting DelaySeconds per message in your send logic, make sure there's no bug accidentally setting it to 60+ seconds (a typo or misconfigured variable could cause this)

5. Check Lambda Concurrency Utilization #

Even with 50 reserved concurrency, Lambda might not be scaling up to process messages if the trigger isn't correctly signaling demand.

• Look at the Lambda ConcurrentExecutions CloudWatch metric: if the maximum value is well below 50, that means Lambda isn't using all available capacity. This could indicate an issue with the SQS-Lambda trigger's scaling logic, or that messages aren't being detected for polling.

6. Rule Out Silent Lambda Hangs #

If your Lambda gets stuck (e.g., on a slow external API call) without timing out, SQS will eventually mark the message as unprocessed after the 120-second visibility timeout. The message will requeue, and its age will continue to accumulate from the original send time.

• Add logging at key points in your Lambda code to track how long each message takes to process. Look for any steps that are taking longer than expected, even if they don't trigger a timeout.

Final Notes #

Keep in mind that ApproximateAgeOfOldestMessage is an approximate metric—it's based on sampling, so occasional spikes might not reflect actual message processing delays. Cross-reference it with NumberOfMessagesReceived and NumberOfMessagesDeleted to confirm if messages are flowing through the queue properly.

Hope these tips help you track down the issue! Let me know if you uncover any specific clues in your logs or metrics.

内容的提问来源于stack exchange，提问作者Patrick W.