基于GKE中Kubernetes容器日志生成Prometheus自定义指标咨询

Hey there! Let's break down how you can set up custom metrics from your GKE container logs (already collected by Stackdriver) using Prometheus. Here's a step-by-step approach tailored to your needs:

Before diving into metrics, take a quick check in the GCP Logging console to ensure you can reliably target your app's logs:

• Filter logs using resource.type="k8s_container" plus your cluster/namespace/container labels to locate the exact log entries you want to analyze.
• Verify that the keyword pattern you want to count is present in fields like textPayload or jsonPayload (depending on how your app formats logs).

This is the quickest path if you prefer leveraging GCP's built-in tools:

• Create a custom log-based counter metric:
  1. Go to GCP Console > Logging > Logs-based Metrics > Create Metric > Counter.
  2. Build a filter to match your target logs, using regex for keyword matching. Example:

     resource.type="k8s_container" 
     AND resource.labels.cluster_name="your-cluster-name"
     AND resource.labels.namespace_name="your-namespace"
     AND resource.labels.container_name="your-app-container"
     AND textPayload=~"your-regex-keyword-pattern"
     

  3. Name your metric (e.g., app_critical_error_logs), add a description, and save it.
• Configure Prometheus to scrape this GCP metric:
  1. Deploy the stackdriver_exporter to your GKE cluster (a lightweight tool that pulls GCP metrics into Prometheus).
  2. Assign a ServiceAccount to the exporter with permissions like monitoring.timeSeries.list and monitoring.metricDescriptors.list.
  3. Add a scrape config to your Prometheus setup to pull metrics from the exporter's endpoint (default port 9255), targeting your custom log-based metric.

This path offers more flexibility if you need complex log processing before generating metrics:

• Deploy Fluent Bit as a DaemonSet in GKE:
  Replace or supplement GKE's default Fluentd setup with Fluent Bit, which can capture container stdout logs locally and process them in real time.
• Configure Fluent Bit filters and Prometheus output:
  1. Use the tail input plugin to read logs from /var/log/containers/*.log.
  2. Add the kubernetes filter to attach cluster/namespace/container metadata to log entries.
  3. Use the grep filter to match your keyword regex, then route matching logs to the prometheus output plugin. Example config snippet:

     [FILTER]
         Name grep
         Match *
         Regex textPayload your-regex-keyword-pattern
     
     [OUTPUT]
         Name prometheus
         Match *
         Host 0.0.0.0
         Port 2021
         Metric_Name app_log_keyword_counter
         Metric_Type counter
         Label_Key kubernetes.container_name
         Label_Key kubernetes.namespace_name
     

• Scrape Fluent Bit's metrics endpoint:
  Add a scrape config in Prometheus to pull metrics from the Fluent Bit DaemonSet service (port 2021, path /metrics).

Once your setup is running:

• Head to the Prometheus UI and search for your custom metric name to confirm data is being collected.
• For visualization, connect Prometheus to Grafana and build dashboards to track trends in your keyword count over time.

内容的提问来源于stack exchange，提问作者samanta