如何排查AWS账单中NAT Gateway数据处理高成本的来源？

Got it, let's break down how to track down exactly what's driving that high NAT Gateway data processed cost. NAT Gateways handle traffic from private subnets to the internet (or AWS services without a VPC endpoint), so the usage ties directly to resources in your private subnets. Here's a step-by-step approach to pinpoint the source:

1. First, Identify the Problematic NAT Gateway #

Start with AWS Cost Explorer to narrow down which NAT Gateway is racking up the costs:

• Open Cost Explorer, filter the service to EC2, then locate the "EC2: NAT Gateway - Data Processed" cost line item.
• Use the Group by dropdown and select Resource—this will show you the specific NAT Gateway ID(s) associated with the high costs. This is critical because you might have multiple NAT Gateways across different VPCs.

2. Track Traffic with VPC Flow Logs #

VPC Flow Logs are your best tool to see exactly which resources are sending traffic through the NAT Gateway:

• Enable Flow Logs for either the NAT Gateway itself (select the NAT Gateway in the VPC console, then choose "Create flow log") or the entire VPC it resides in. Send logs to CloudWatch Logs for easier querying.
• Once logs start flowing, use CloudWatch Logs Insights to run a query like this (replace eni-xxxxxx with your NAT Gateway's Elastic Network Interface ID):

fields srcAddr, dstAddr, bytes, action, logStream
| filter interfaceId = "eni-xxxxxx"
| sort bytes desc
| limit 20

This will return the top traffic sources by byte count. The srcAddr is the private IP of the resource in your private subnet sending traffic through the NAT Gateway.

3. Map Private IPs to Specific Resources #

Now that you have the private IPs from Flow Logs, match them to actual resources:

• EC2 Instances: Head to the EC2 console, go to the Instances page, and use the search bar to filter by private IP. This will directly show you which instance is generating the traffic.
• EMR Clusters: EMR nodes (core/task) live in private subnets—check the EMR console's cluster details, then look at the instance list to match the private IPs from your Flow Logs.
• S3 Traffic: Note that S3 itself doesn't send traffic through NAT Gateways; instead, traffic to S3 via NAT Gateway comes from private subnet resources accessing S3 without a VPC endpoint. So the source will still be the EC2/EMR/etc. instances making those S3 calls.

4. Tie Traffic to IAM Users/Roles (If Needed) #

If you want to know which IAM identity is associated with the traffic (e.g., an instance role used to call AWS services), use AWS CloudTrail:

• Open CloudTrail, filter events by the time range where costs spiked, and look for API calls matching the traffic pattern (e.g., s3:GetObject, ec2:DescribeInstances).
• Check the sourceIPAddress field to match the private IP from Flow Logs, then look at userIdentity to see which IAM role or user is linked to that resource.

5. Look for Unintended Traffic Patterns #

Don't forget to check for unexpected usage:

• Is an EC2 instance running a backup job that's syncing large amounts of data to an external service?
• Does an EMR cluster have a batch job that's pulling/pushing data across regions without a VPC endpoint?
• Could there be misconfigured resources (like a cron job) that's generating continuous high traffic?

Pro tip: If you're accessing AWS services like S3 from private subnets, set up VPC endpoints for those services. This lets traffic go directly between your VPC and the service without routing through the NAT Gateway, which will cut down on data processed costs significantly.

内容的提问来源于stack exchange，提问作者Jaishree Mishra