如何将Fluentd收集的日志转发至GCP Cloud Storage

Prerequisites #

• GCP Cloud Storage bucket created
• GCP Service Account with roles/storage.objectCreator permission assigned to the bucket
• Service Account JSON key file downloaded locally

Step 1: Install the Fluentd GCS Plugin #

Use the appropriate command based on your Fluentd distribution:

• For vanilla Fluentd:

  gem install fluent-plugin-gcs
  

• For td-agent (Treasure Data's Fluentd distribution):

  td-agent-gem install fluent-plugin-gcs
  

Step 2: Set Up GCP Authentication #

Point Fluentd to your service account key file via environment variable:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your-service-account-key.json"

If running Fluentd as a system service, add this variable to your service configuration (e.g., systemd unit file for td-agent).

Step 3: Modify fluentd.conf #

Update your configuration to add an output block for GCP Cloud Storage. You can retain the existing Loki output for dual delivery, or replace it if you only want logs sent to GCS.

Example Configuration Snippet #

# Existing Docker source (keep this)
<source>
  @type docker
  tag docker.*
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/fluentd/docker.pos
  read_from_head true
</source>

# Optional existing filters (retain if you use them)
<filter docker.**>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

# Existing Loki output (keep for dual delivery)
<match docker.**>
  @type loki
  url "http://loki:3100/loki/api/v1/push"
  <label>
    job "docker"
    container_name ${tag_parts[2]}
  </label>
</match>

# New GCS output block
<match docker.**>
  @type gcs

  project_id your-gcp-project-id
  bucket your-bucket-name
  path logs/%Y/%m/%d/
  format json
  compression gzip
  buffer_chunk_limit 100MB
  buffer_queue_limit 10
  flush_interval 10m
  timezone UTC

  # Uncomment below if not using the environment variable for credentials
  # credentials /path/to/your-service-account-key.json
</match>

Key Configuration Details #

• project_id: Your GCP project ID where the bucket is hosted
• bucket: Exact name of your Cloud Storage bucket
• path: Chronological directory structure using time variables (e.g., %Y for year, %m for month) to organize logs
• format: Output format for logs (json is recommended for structured data)
• compression: gzip reduces storage and bandwidth costs
• buffer settings: Adjust based on your log volume to balance upload frequency and file size

Step 4: Restart Fluentd #

Apply changes by restarting the service:

# For td-agent
sudo systemctl restart td-agent

# For vanilla Fluentd
sudo systemctl restart fluentd

Step 5: Verify Log Delivery #

Check your GCP Cloud Storage bucket after the configured flush_interval—you should see compressed log files in the specified path.

内容的提问来源于stack exchange，提问作者Aditya Sahu