如何使用CloudFormation为多个AWS CloudWatch Log Groups创建Subscription Filters？能否通过单条YAML配置实现多Log Groups及全量Log Groups订阅？

Great question! Let's break this down into the two parts of your requirement and walk through how to approach it in CloudFormation:

1. Creating subscription filters for multiple specific Log Groups #

You can use CloudFormation's native features to generate multiple subscription filters from a single configuration pattern, avoiding the need to write individual entries for each log group. The key here is leveraging the Fn::ForEach intrinsic function (available in CloudFormation modules or with the latest template capabilities) to loop through a predefined list of log groups.

Here's a simplified working example:

Parameters:
  TargetLambdaArn:
    Type: String
    Description: ARN of the Lambda function to route logs to
  SpecificLogGroups:
    Type: List<String>
    Description: List of specific log group names to attach filters to

Resources:
  # Dynamically generate a subscription filter for each log group in the list
  Fn::ForEach::LogGroupLoop:
    - LogGroupName
    - !Ref SpecificLogGroups
    - ${LogGroupName}SubscriptionFilter:
        Type: AWS::Logs::SubscriptionFilter
        Properties:
          LogGroupName: !Ref LogGroupName
          FilterPattern: "" # Adjust your desired filter pattern here
          DestinationArn: !Ref TargetLambdaArn
          Distribution: ByLogStream

This block will automatically create a separate AWS::Logs::SubscriptionFilter resource for every log group in your SpecificLogGroups list—all from one configuration pattern.

2. Creating a single subscription filter for ALL Log Groups #

This is where native CloudFormation has a limitation: the AWS::Logs::SubscriptionFilter resource requires an explicit LogGroupName value, with no built-in support for wildcards or automatically targeting every existing log group (including those not created by your stack).

To achieve a global subscription filter, you'll need to use a Lambda-backed custom resource. This Lambda function can:

• Iterate over all log groups in your account/region
• Create or update a subscription filter for each one pointing to your target destination
• Clean up filters if the stack is deleted

Here's a high-level template example for this setup:

Resources:
  GlobalSubscriptionFilterLambda:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.12
      Handler: index.lambda_handler
      Code:
        ZipFile: |
          import boto3
          import cfnresponse

          logs_client = boto3.client('logs')

          def lambda_handler(event, context):
              try:
                  target_arn = event['ResourceProperties']['TargetArn']
                  filter_pattern = event['ResourceProperties'].get('FilterPattern', '')
                  
                  # Fetch all log groups using pagination
                  paginator = logs_client.get_paginator('describe_log_groups')
                  for page in paginator.paginate():
                      for log_group in page['logGroups']:
                          log_group_name = log_group['logGroupName']
                          # Create or update the subscription filter
                          logs_client.put_subscription_filter(
                              logGroupName=log_group_name,
                              filterName='GlobalSubscriptionFilter',
                              filterPattern=filter_pattern,
                              destinationArn=target_arn
                          )
                  cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
              except Exception as e:
                  cfnresponse.send(event, context, cfnresponse.FAILED, {'Error': str(e)})

  GlobalSubscriptionFilter:
    Type: Custom::GlobalSubscriptionFilter
    Properties:
      ServiceToken: !GetAtt GlobalSubscriptionFilterLambda.Arn
      TargetArn: !Ref TargetLambdaArn
      FilterPattern: "" # Adjust your filter pattern here

Note: This Lambda will only process existing log groups at stack creation/update time. If new log groups are added later, you'll need to trigger the custom resource again (e.g., by updating the stack with a dummy parameter change) or add event-driven logic (like CloudTrail triggers) to handle new log groups automatically.

Final Verdict #

You can't handle both requirements with a single CloudFormation resource entry, but you can combine the two patterns above in one template to meet both goals: dynamic per-log-group filters for specific groups, and a custom resource to manage filters for all log groups.

内容的提问来源于stack exchange，提问作者Tomasz Szymanek