Docker日志写入前过滤及直接将指定错误日志发送至GCP的方法咨询

Great question! Ditching unnecessary middlemen like Fluentd makes total sense here—let’s break down two direct, lightweight approaches to get your level='error' logs into GCP without extra hops:

Option 1: Docker Native Google Cloud Logging Driver + GCP Log Routing #

This approach uses Docker’s built-in google-cloud-logging driver to send logs directly to GCP Cloud Logging, then leverages GCP’s native filtering to isolate error logs.

Steps: #

1. Configure Docker to use the GCP Log Driver

   • Either set it as the default by editing /etc/docker/daemon.json:

     {
       "log-driver": "google-cloud-logging",
       "log-opts": {
         "gcp-project": "your-gcp-project-id"
       }
     }
     

   • Or specify it per container when launching:

     docker run --log-driver=google-cloud-logging --log-opt gcp-project=your-gcp-project-id your-image
     

   Note: Your host needs the GCP service account credentials (via GOOGLE_APPLICATION_CREDENTIALS env var or metadata server if running on GCE/GKE).

2. Set Up GCP Log Routing for Error Filtering

   • Head to the GCP Cloud Logging console, go to Logs Router
   • Create a new route:
     • For Filter, use a query that matches your error logs:
       • If your logs are JSON-formatted: jsonPayload.level = "error"
       • If they’re plain text: textPayload: "level='error'"
     • Choose a destination (e.g., a dedicated log bucket, Pub/Sub topic for alerts, or Cloud Monitoring)

Pros: No extra software to install; filtering is managed centrally in GCP. Cons: All logs are sent to GCP first (you might incur minor extra costs for non-error logs).

Option 2: GCP Ops Agent (Local Filtering + Direct Shipping) #

If you want to filter logs before sending them to GCP (to save bandwidth/cost), use the GCP Ops Agent—Google’s lightweight tool for collecting logs/metrics, which supports local filtering.

Steps: #

1. Install the Ops Agent on your Docker host
   Follow GCP’s official installation steps for your OS (it’s a simple package install).

2. Configure the Agent to Filter Docker Logs
   Edit the Ops Agent config at /etc/google-cloud-ops-agent/config.yaml:

   logging:
     receivers:
       docker:
         type: docker
         include_paths:
           - /var/lib/docker/containers/*/*.log
         exclude_paths: []
         docker_labels: true
     processors:
       filter_errors:
         type: filter
         expr: 'textPayload contains "level=''error''" OR jsonPayload.level == "error"'
     exporters:
       google_cloud_logging:
         type: google_cloud_logging
     service:
       pipelines:
         docker_errors:
           receivers: [docker]
           processors: [filter_errors]
           exporters: [google_cloud_logging]
   

   • The filter_errors processor ensures only logs matching your error condition are sent to GCP.
   • Restart the agent: sudo systemctl restart google-cloud-ops-agent

Pros: Filters logs locally, reducing data sent to GCP; more flexible filtering logic. Cons: Requires installing the Ops Agent on each host.

Key Notes #

• Make sure your logs are structured (JSON) if possible—it makes filtering far more reliable than plain text matching.
• Verify IAM permissions: Both approaches need the host/agent to have the roles/logging.logWriter permission in GCP.

内容的提问来源于stack exchange，提问作者meotimdihia