如何将API网关访问日志存储到单个LOG STREAM并自定义日志流？

Great question! I’ve hit this exact limitation with API Gateway before—out of the box, you can only specify a CloudWatch Logs log group, and API Gateway automatically creates log streams based on its own internal logic (like API ID, stage, and instance identifiers). IAM policies aren’t the solution here, but there are two solid workarounds to get full control over your log streams:

1. Use CloudWatch Logs Subscription Filters + Lambda to Redirect Logs #

This is the most flexible approach, letting you route logs to custom streams based on any criteria you want (request path, client IP, API key, etc.):

• Step 1: First, configure API Gateway to send access logs to a temporary "staging" log group (this is mandatory, since API Gateway needs a default target).
• Step 2: Create a Lambda function that will handle log routing. The function will:
  1. Parse the incoming CloudWatch Logs event payload (you’ll need to decode the base64-encoded log data).
  2. Extract the metadata you want to use to split logs into streams (e.g., $context.path from your custom log format, or request ID).
  3. Define a custom log stream name (e.g., api-prod-checkout for all /checkout requests, or client-192.168.1.1 for a specific IP).
  4. Use the CloudWatch Logs PutLogEvents API to send the log entry to your target log stream (create the stream first with CreateLogStream if it doesn’t exist).
• Step 3: Set up a CloudWatch Logs Subscription Filter on your staging log group, pointing it to your Lambda function.
• Step 4: Attach an IAM policy to your Lambda execution role that allows:
  • logs:PutLogEvents and logs:CreateLogStream on your target log group
  • logs:DescribeLogStreams (optional, to check if streams exist before creating)
  • logs:FilterLogEvents on your staging log group

2. Leverage API Gateway Stage Variables + Multiple Log Groups (Simpler Alternative) #

If you just need to split logs by stage or a fixed dimension, you can use stage variables to route logs to different log groups, and then let CloudWatch handle streams within each group:

• Create multiple log groups (e.g., api-prod-checkout, api-prod-users).
• Define a stage variable in your API Gateway stage that specifies the target log group based on your routing rule.
• Use the stage variable in your API Gateway access log configuration (e.g., $(stageVariables.targetLogGroup)).
• Note: This only works if your routing logic is tied to stage-level or deployment-level variables—you can’t dynamically split based on request attributes with this method.

Important Note About IAM Policies #

Your initial thought about IAM is understandable, but IAM can’t force API Gateway to specify custom log streams. IAM only controls which log groups API Gateway can write to, not how it structures streams within those groups. The limitation is baked into API Gateway’s log configuration API, so workarounds like the ones above are necessary.

内容的提问来源于stack exchange，提问作者Newbie