cAdvisor无法显示所有容器运行时长问题排查（Prometheus+Grafana环境）

Let’s walk through the possible issues and fixes based on your setup:

1. Prometheus Target Configuration Check #

First, verify if Prometheus is correctly scraping all metrics from cAdvisor:

• Check scrape targets: Go to Prometheus's web UI (via your Apache proxy if needed) and navigate to the Status > Targets page. Ensure the cAdvisor target (172.28.5.201:8080) shows as UP and that the scrape is successful.
• Scrape interval & relabeling: Confirm your Prometheus config doesn’t have relabel rules that filter out metrics from non-cAdvisor containers. Double-check that the scrape_interval is set to a reasonable value (e.g., 15s) so metrics are pulled frequently enough.
• Proxy impact: Since you have an Apache reverse proxy, make sure Prometheus is scraping cAdvisor directly via its internal IP (172.28.5.201:8080) instead of going through the proxy. If it uses the proxy, ensure the proxy doesn’t block or modify the metric endpoints (like /metrics).

2. cAdvisor Deployment Validation #

Your cAdvisor run command looks mostly complete, but let’s rule out deployment issues:

• Check cAdvisor’s local UI: Access cAdvisor directly at http://172.28.5.201:8080 from within your network. If it only shows its own container’s uptime here, the problem is with cAdvisor’s data collection, not Prometheus/Grafana.
• Inspect cAdvisor logs: Run this command to check for errors:

  sudo docker logs cadvisor
  

  Look for warnings about missing permissions, failed directory access, or issues reading container metadata from /var/lib/docker or /cgroup.
• Verify mount points: Ensure the mounted volumes are accessible inside the cAdvisor container:

  sudo docker exec -it cadvisor ls /var/lib/docker/containers
  

  If this returns an empty list or permission errors, the /var/lib/docker mount might be misconfigured (though your command uses :ro which should be fine, double-check host-side permissions).

3. Grafana Query Debugging #

If cAdvisor and Prometheus have the data but Grafana doesn’t display it, focus on the query:

• Validate the metric exists in Prometheus: Go to Prometheus’s Graph page and search for container_start_time_seconds. You should see entries for all your containers (Prometheus, Grafana, cAdvisor). If only cAdvisor appears here, the issue is upstream in Prometheus/cAdvisor.
• Use a working uptime query: Try this standard query to calculate container uptime (in seconds):

  time() - container_start_time_seconds{container_name!="", container_name!="POD"}
  

  Adjust the labels (like container_name) if your setup uses different label keys (e.g., name instead of container_name). If you get "no datapoints", confirm the label filters aren’t excluding the Prometheus/Grafana containers.

Additional Notes #

• Network mode: Your cAdvisor uses a custom network (docker8443). Ensure Prometheus is attached to the same network, or can reach the static IP 172.28.5.201 across networks.
• Privileged mode: You already enabled --privileged=true, which is good for cAdvisor to access cgroup data, but double-check that the /cgroup mount is working correctly inside the container.

内容的提问来源于stack exchange，提问作者Sean Hu