cAdvisor Docker服务未向Prometheus发布name与image标签求助

我是监控领域纯新手，已通过docker-compose部署cAdvisor、Prometheus、Grafana服务，期望通过Prometheus查询cAdvisor指标，在Grafana中展示容器运行状态。但参考示例使用类似container_last_seen{image="blah"}或container_last_seen{name="blah"}的查询语句时无法生效，原因是cAdvisor输出的metrics中不包含name和image标签。

docker-compose.yml配置 #

version: "3.9"

services:
  backend:
    [... hide this code for privacy]
  frontend:
    [... hide this code for privacy]
  db:
    image: arm64v8/mysql:8.0
    [... hide this code for privacy]

  prometheus:
    image: prom/prometheus:v2.46.0
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/etc/prometheus/console_libraries"
      - "--web.console.templates=/etc/prometheus/consoles"
      - "--storage.tsdb.retention.time=200h"
      - "--web.enable-lifecycle"
      - "--web.enable-remote-write-receiver"
      - "--enable-feature=native-histograms"
    restart: unless-stopped
    expose:
      - 9090
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"
    user: root

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.1
    container_name: cadvisor
    privileged: true
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      #- /cgroup:/cgroup:ro #doesn't work on MacOS only for Linux
    restart: unless-stopped
    expose:
      - 8080
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    ports:
      - "3100:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
    restart: unless-stopped
    expose:
      - 3100
    networks:
      - monitor-net

networks:
  monitor-net:
    driver: bridge

volumes:
  dev-db-data:
  dev-static-data:
  grafana_data: {}
  prometheus_data: {}

prometheus.yml配置 #

global:
  scrape_interval:     15s
  evaluation_interval: 15s

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'docker-host-alpha'

# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
  - job_name: 'nodeexporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['nodeexporter:9100']

  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'pushgateway'
    scrape_interval: 10s
    honor_labels: true
    static_configs:
      - targets: ['pushgateway:9091']

cAdvisor metrics示例（无name/image标签） #

# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="7dfeea7d",cadvisorVersion="v0.47.1",dockerVersion="",kernelVersion="5.15.49-linuxkit-pr",osVersion="Alpine Linux v3.16"} 1
# HELP container_blkio_device_usage_total Blkio Device bytes usage
# TYPE container_blkio_device_usage_total counter
container_blkio_device_usage_total{device="",id="/013-mount-services",major="7",minor="0",operation="Read"} 53760 1695050571700
container_blkio_device_usage_total{device="",id="/013-mount-services",major="7",minor="0",operation="Write"} 0 1695050571700
container_blkio_device_usage_total{device="",id="/artifactory",major="7",minor="0",operation="Read"} 1.250304e+07 1695050574896
container_blkio_device_usage_total{device="",id="/artifactory",major="7",minor="0",operation="Write"} 0 1695050574896
container_blkio_device_usage_total{device="",id="/binfmt",major="7",minor="0",operation="Read"} 2.6136576e+07 1695050572835
container_blkio_device_usage_total{device="",id="/binfmt",major="7",minor="0",operation="Write"} 0 1695050572835
container_blkio_device_usage_total{device="",id="/container-filesystem",major="7",minor="0",operation="Read"} 1.4364672e+07 1695050549973
container_blkio_device_usage_total{device="",id="/container-filesystem",major="7",minor="0",operation="Write"} 0 1695050549973
container_blkio_device_usage_total{device="",id="/devenv-service",major="7",minor="0",operation="Read"} 5.885952e+06 1695050574243
container_blkio_device_usage_total{device="",id="/devenv-service",major="7",minor="0",operation="Write"} 0 1695050574243
container_blkio_device_usage_total{device="",id="/dhcpcd",major="7",minor="0",operation="Read"} 1.83296e+06 1695050559325

解决思路 #

• 添加Docker Socket挂载：cAdvisor需要访问Docker daemon才能获取容器的name、image等元数据。在docker-compose.yml的cadvisor服务volumes中新增一行：

  - /var/run/docker.sock:/var/run/docker.sock:ro
  

• 重启cAdvisor服务：执行docker-compose restart cadvisor，等待1-2分钟后重新访问cAdvisor的/metrics端点，检查是否出现name和image标签。
• 临时用id标签查询：如果还没生效，可以先用现有id标签做查询，比如container_last_seen{id=~".*devenv-service.*"}，先验证数据可查。
• 排查cAdvisor日志：执行docker logs cadvisor，查看是否有无法访问Docker socket的报错，确认权限或路径问题。

内容的提问来源于stack exchange，提问作者Ignacio Borrajo Iglesias