Istio遥测流量未在Grafana和Kiali正常显示求助

Hey there! Let's walk through why you're not seeing traffic data in Kiali and Grafana even though Prometheus has trace info. Since your setup is close to the Bookinfo tutorial, let's break down the most common checks and fixes:

1. First, Confirm Istio Sidecar Injection is Working #

This is the #1 culprit for missing metrics—if your pods don't have the Istio proxy sidecar, no traffic data gets collected.

• Run this command to check your pods:

  kubectl get pods -n <your-namespace>
  

  Each pod should show 2/2 containers ready (your app container + istio-proxy). If you only see 1/1, the sidecar wasn't injected.
• Fixes for missing injection:
  • Make sure your namespace has auto-injection enabled:

    kubectl label namespace <your-namespace> istio-injection=enabled --show-labels
    

  • Or add this annotation directly to your Deployment's metadata.annotations section:

    sidecar.istio.io/inject: "true"
    

  • Redeploy your app after making these changes.

2. Verify Istio Telemetry is Enabled #

Istio's default telemetry config generates the metrics Kiali/Grafana rely on—if it's disabled or modified, metrics won't flow.

• Check the default telemetry resource in the istio-system namespace:

  kubectl get telemetry -n istio-system
  

• Inspect the details to ensure HTTP metrics are enabled:

  kubectl describe telemetry default -n istio-system
  

  Look for sections referencing metrics like request_total, request_duration, and request_bytes—these should be included in the default config.

3. Double-Check Service & Pod Label Matching #

Kiali uses labels to map services to pods, so mismatched labels break metric aggregation:

• Verify your Service's selector correctly targets your pods:

  kubectl describe svc <your-service-name>-svc -n <your-namespace>
  

  Under Endpoints, you should see the IPs of your running pods. If it's empty, your selector labels don't match the pod labels.
• From your Helm config, your Service uses app: {{ .Values.service.name }} and your pods have the same app label—this looks correct, but confirm the rendered labels match (use kubectl describe pod <pod-name> to check).

4. Fix Grafana Data Source & Panel Queries #

Even if Prometheus has data, Grafana might be looking in the wrong place:

• In Grafana, go to Configuration > Data Sources and confirm your Prometheus data source points to the correct instance (usually http://prometheus.istio-system.svc.cluster.local:9090).
• Open your Istio traffic panel and check the queries. For example, a query like istio_requests_total{namespace="istio-system"} will only show data from the istio-system namespace—update it to your project namespace, or remove the namespace filter entirely.

5. Adjust Kiali View Settings #

Sometimes it's just a UI setting issue:

• In Kiali, switch to your project namespace using the dropdown at the top of the page.
• Set the time range to something recent (e.g., Last 5 Minutes)—old time ranges might not include your test traffic.
• In the Graph view, make sure the Traffic toggle is enabled, and you've selected a relevant metric like Request Count.

6. Validate Prometheus Metric Labels #

Since you can see trace data in Prometheus, check if Istio's metrics exist and have the right labels:

• In Prometheus, run the query:

  istio_requests_total
  

  Look for results with destination_service="<your-service-name>-svc.<your-namespace>.svc.cluster.local". If you don't see these, the sidecar isn't generating metrics—go back to step 1 to fix sidecar injection.

Quick Notes on Your Config #

• Your Gateway and VirtualService look solid—they route traffic to your service correctly. Test traffic with curl http://<istio-ingressgateway-ip>/ to ensure requests are hitting your app.
• Your Deployment uses apps/v1beta2, which is deprecated. It won't break metrics, but consider upgrading to apps/v1 for long-term compatibility.

内容的提问来源于stack exchange，提问作者Soufiaane