AWS CloudFormation中ECS容器能否配置多个LogDriver以同时推送日志至Elasticsearch与Loki

Great question! Unfortunately, AWS ECS doesn't support setting up multiple LogDrivers for a single container directly—you’re limited to one LogDriver per container definition at any given time. But don’t worry, there are several solid workarounds to achieve your goal of sending logs to both Elasticsearch and Loki. Let’s break them down:

Option 1: Update Your FluentD Configuration to Send to Both Destinations #

Instead of switching to json-file, keep using the fluentd LogDriver and modify your FluentD instance to route logs to both Elasticsearch and Loki at the same time. This is probably the simplest approach if you already have FluentD running alongside your ECS tasks.

You’ll need to add Loki’s output plugin to your FluentD configuration file. Here’s a quick example of what that might look like:

# Existing input (collecting logs from ECS via fluentd driver)
<source>
  @type forward
  port 24224
</source>

# Route logs to both Elasticsearch and Loki
<match docker.container.green>
  # Send to Elasticsearch
  @type elasticsearch
  host your-es-host
  port 9200
  index_name ecs-logs-%Y.%m.%d

  # Send to Loki (requires the fluent-plugin-loki gem installed)
  <store>
    @type loki
    url http://your-loki-host:3100/loki/api/v1/push
    label_keys ["tag"]
    flush_interval 10s
  </store>
</match>

Then, revert your ECS task definition back to using the fluentd LogDriver (uncomment your original fluentd config snippet) and restart your tasks. FluentD will handle pushing logs to both services simultaneously.

Option 2: Stick with json-file + Sidecar Container for Dual Log Collection #

If you prefer keeping json-file as your primary LogDriver, deploy a sidecar container (like Promtail or FluentD) alongside your green-container to read the log files and send them to both destinations.

Example with Promtail: #

1. Keep your current json-file LogDriver configuration in the task definition.
2. Add a Promtail sidecar container to the task, mounting the ECS container log directory (typically /var/log/containers/ on EC2 instances; for Fargate, use the appropriate volume mounts for container logs).
3. Configure Promtail to scrape the json-file logs and send them to Loki. For Elasticsearch, you can either add a FluentD sidecar to scrape the same logs and forward them to ES, or use a tool like Grafana Loki’s export feature to route logs to ES.

Option 3: Use AWS FireLens (FluentBit-Based) for ECS-Native Multi-Target Routing #

AWS FireLens is an ECS-native log routing solution built on FluentBit, which supports sending logs to multiple destinations out of the box. This is the most AWS-integrated approach and works smoothly with both EC2 and Fargate launch types.

Here’s how you’d adjust your task definition to use FireLens:

GreenTaskDefinition:
  Type: 'AWS::ECS::TaskDefinition'
  Properties:
    ContainerDefinitions:
      - Name: green-container
        LogConfiguration:
          LogDriver: awsfirelens
          Options:
            Name: awsfirelens
          FireLensConfiguration:
            Type: fluentbit
            Options:
              enable-ecs-log-metadata: "true"
      # FireLens sidecar (managed by ECS)
      - Name: firelens-log-router
        Image: 906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable
        Essential: true
        ConfigFiles:
          - Value: |
              [INPUT]
                  Name              forward
                  Port              24224

              [OUTPUT]
                  Name              elasticsearch
                  Match             *
                  Host              your-es-host
                  Port              9200
                  Index             ecs-logs-%Y.%m.%d

              [OUTPUT]
                  Name              loki
                  Match             *
                  Host              your-loki-host
                  Port              3100
                  URI               /loki/api/v1/push
                  Labels            job=ecs-green-container,tag=green
            Name: fluent-bit.conf

FireLens will collect logs from your main container and route them to both Elasticsearch and Loki according to your FluentBit configuration.

Each approach has its pros: Option 1 is best if you already have FluentD set up, Option 3 is ideal for a fully integrated AWS workflow, and Option 2 gives you flexibility if you want to stick with json-file.

内容的提问来源于stack exchange，提问作者Ken Tsoi