如何在Kubernetes中配置实时获取应用请求平均延迟（30秒/1分钟）

Great question! That 210-second delay from Stackdriver's load balancer metrics is way too slow for immediate latency insights. Below is a fully app-agnostic solution (no Java code changes needed) that works for any deployed application in your cluster, delivering latency data every 30 seconds or 1 minute.

Core Approach #

We'll use a sidecar proxy pattern to intercept all traffic to your app, collect detailed latency metrics via the proxy, then use Prometheus for fast scraping and Grafana for visualization. This keeps your application code untouched and works for any language/framework.

Step 1: Inject a Metrics-Enabled Sidecar Proxy #

We'll use Nginx (with a Prometheus exporter) as the sidecar—lightweight, easy to configure, and widely supported. Alternatively, you can use Envoy for more advanced traffic management, but Nginx is perfect for most use cases.

Example Deployment with Sidecar #

Add these containers to your existing Deployment (replace placeholders with your app details):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: java-app
    spec:
      containers:
        # Your existing Java app container
        - name: java-app
          image: your-java-app-image:latest
          ports:
            - containerPort: 8080
        # Nginx sidecar proxy (routes all traffic to your app)
        - name: nginx-proxy
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
        # Nginx Prometheus Exporter (scrapes proxy metrics)
        - name: nginx-exporter
          image: nginxinc/nginx-prometheus-exporter:latest
          args:
            - -nginx.scrape-uri=http://localhost:8080/stub_status
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-proxy-config

Nginx ConfigMap (Proxy + Metrics Setup) #

Create this ConfigMap to configure Nginx as a reverse proxy and enable metrics scraping:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-proxy-config
data:
  nginx.conf: |
    events {}
    http {
      server {
        listen 80;
        # Route all traffic to your Java app
        location / {
          proxy_pass http://localhost:8080;
          proxy_set_header X-Real-IP $remote_addr;
        }
        # Enable stub_status for exporter to collect metrics
        location /stub_status {
          stub_status on;
          allow 127.0.0.1;
          deny all;
        }
      }
    }

Step 2: Configure Prometheus for Fast Scraping #

Set up Prometheus to scrape the Nginx exporter every 30 seconds. Use a ServiceMonitor (if you're using the Prometheus Operator) to auto-discover your app pods.

Example ServiceMonitor #

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: java-app-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: java-app
  endpoints:
    - port: metrics
      interval: 30s  # Scrape every 30 seconds

Don't forget to expose the exporter's metrics port (default 9113) via a Kubernetes Service for Prometheus to access.

Step 3: Visualize Latency in Grafana #

Import the official Nginx Prometheus Dashboard (or build your own) to view average latency split by GET/POST requests. Use these PromQL queries to get exactly what you need:

# Average GET request latency (seconds)
avg(nginx_http_request_duration_seconds_sum{method="GET"} / nginx_http_request_duration_seconds_count{method="GET"}) by (pod)

# Average POST request latency (seconds)
avg(nginx_http_request_duration_seconds_sum{method="POST"} / nginx_http_request_duration_seconds_count{method="POST"}) by (pod)

You can adjust the Grafana panel refresh rate to match your scraping interval (30s/1min) for near-real-time insights.

Alternative: Envoy Sidecar for Advanced Use Cases #

If you need more granular traffic control (like rate limiting, circuit breaking), swap Nginx for Envoy. Envoy natively exposes Prometheus metrics with detailed latency breakdowns, and you can configure scraping intervals the same way. For a managed approach, consider Istio, but manual Envoy injection works well for simpler setups.

Key Benefits #

• App-Agnostic: No changes to your Java app (or any other app) required—works for all languages.
• Low Latency: 30s/1min scraping intervals deliver immediate latency data.
• Scalable: Roll this out to all your Kubernetes deployments with consistent sidecar configurations.

内容的提问来源于stack exchange，提问作者anushiya-thevapalan