AWS CloudWatch跨账号日志收集：基于EC2实例配置文件的需求

Alright, let's walk through exactly how to set up this centralized logging flow based on your requirements—no Kinesis, only the core red-labeled components, and tying everything to your on-prem ELK stack:

1. Cross-Account CloudWatch Logs Permissions (Core Component) #

First, we need to build the trust and permissions between your child accounts and the central logging account to let logs flow into a single CloudWatch repository.

Central Account Setup #

Create an IAM role (e.g., CrossAccountLogDeliveryRole) in your central account with two key parts:

• Trust Policy: Grants your child accounts permission to assume this role

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::CHILD_ACCOUNT_ID_1:root",
            "arn:aws:iam::CHILD_ACCOUNT_ID_2:root"
            // Add all child account IDs here
          ]
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }
  

• Permissions Policy: Lets the role write logs to your central CloudWatch log group

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "logs:PutLogEvents",
          "logs:CreateLogStream"
        ],
        "Resource": "arn:aws:logs:CENTRAL_REGION:CENTRAL_ACCOUNT_ID:log-group:/centralized-logs/*"
        // Target log group in your central account
      }
    ]
  }
  

Child Account Setup #

For each child account:

• Update the existing EC2 Instance Profile (or the role your logs use) to add permission to assume the central account's CrossAccountLogDeliveryRole:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::CENTRAL_ACCOUNT_ID:role/CrossAccountLogDeliveryRole"
      }
    ]
  }
  

• Create a Subscription Filter on each child account log group you want to aggregate. Set the target to your central account's log group, and select the cross-account role you created. This will auto-forward logs to the central repository.

2. Forward Central CloudWatch Logs to On-Prem ELK (No Kinesis) #

Since you're skipping Kinesis, you have two solid options depending on your real-time needs:

Option 1: Lambda Real-Time Forwarding (Low Latency) #

• Spin up a Lambda function in your central account, set its trigger to your centralized CloudWatch log group (triggers on new log events).
• The Lambda code needs to:
  1. Decode the base64-encoded CloudWatch log events
  2. Parse/transform the log format to match what your ELK stack expects
  3. Send the logs to your on-prem ELK endpoint (either directly to Elasticsearch via its Bulk API, or to Logstash for further processing)
• Make sure the Lambda has network access to your on-prem ELK: if ELK is in a private network, use AWS VPN/Direct Connect to link your AWS VPC to your local network, or configure a public endpoint with proper security.

Option 2: S3 Export + Filebeat (Cost-Effective, Near-Real-Time) #

• Set up a CloudWatch Logs Export Task on your central log group to auto-export logs to an S3 bucket at your desired interval (e.g., every 15 minutes).
• Deploy Filebeat on your local network, configure it to pull log files from the S3 bucket, and send them to your ELK stack.
• This is great for high-volume logs where real-time isn't critical, and it's cheaper than running Lambda 24/7.

3. Critical Things to Keep in Mind #

• Permission Validation: Double-check cross-account role trusts and resource scopes—misconfigurations here are the #1 cause of log flow failures.
• Log Parsing: CloudWatch logs might be in raw text or JSON; add parsing logic in Lambda or Logstash to extract usable fields for ELK.
• Network Security: If exposing ELK to the public, use API keys or IP whitelisting. For private networks, ensure AWS and your local network have stable connectivity.
• Monitoring: Set up CloudWatch Alarms in the central account to alert you if log forwarding success rates drop—this prevents silent log loss.

内容的提问来源于stack exchange，提问作者skohrs