Docker Compose下Prometheus连接cAdvisor被拒绝问题排查与解决

问题现象 #

• cAdvisor的DNS解析正常，Prometheus可ping通cAdvisor容器，但通过telnet访问其私有IP或容器名称时连接被拒绝
• 若允许cAdvisor公网访问，Prometheus能通过公网IP:8081连接成功，但需求是不暴露cAdvisor到公网

相关配置 #

Docker Compose配置 #

version: '3.7'
networks:
  monitoring:
        driver: bridge
volumes:
  prometheus_data: {}
services:
  prometheus:
        image: prom/prometheus:latest
        user: "1000"
        environment:
          - PUID=1000
          - PGID=1000
        container_name: prometheus
        restart: unless-stopped
        volumes:
          - /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
          - /home/sammantar/sammantar/Project/OpenSourceMonitoring/promgrafnode/prometheus:/prometheus
        command:
          - '--config.file=/etc/prometheus/prometheus.yml'
          - '--storage.tsdb.path=/prometheus'
          - '--web.console.libraries=/etc/prometheus/console_libraries'
          - '--web.console.templates=/etc/prometheus/consoles'
          - '--web.enable-lifecycle'
        ports:
          - "9090:9090"
        networks:
          - monitoring


  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "127.0.0.1:8081:8080"
#    network_mode: "host" 
    networks:
      - monitoring
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
      - redis
    deploy:
      mode: global
    networks:
      - monitoring

初始Prometheus采集配置 #

global:
  scrape_interval: 1m

scrape_configs:
  - job_name: "prometheus" (Working)
    scrape_interval: 1m
    static_configs:
    - targets: ["localhost:9090"]

  - job_name: "node" (Working)
    static_configs:
    - targets: ["node-exporter:9100"]

  - job_name: "cadvisor" (not working)
    scrape_interval: 5s
    static_configs:
    - targets: ["cadvisor:8081"]

排查与解决方案 #

尝试使用cAdvisor容器的私有IP、容器别名连接均被拒绝，最终定位问题核心：容器间通信需使用容器内部监听的端口，而非映射到主机的端口。

cAdvisor容器内部监听的端口是8080，ports: "127.0.0.1:8081:8080"只是将主机的8081端口映射到容器的8080端口，该映射仅适用于主机与容器的通信，容器间直接通信需使用容器内部端口。

修正后的Prometheus采集配置：

global:
  scrape_interval: 1m

scrape_configs:
  - job_name: "prometheus" (Working)
    scrape_interval: 1m
    static_configs:
    - targets: ["localhost:9090"]

  - job_name: "node" (Working)
    static_configs:
    - targets: ["node-exporter:9100"]

  - job_name: "cadvisor" (not working)
    scrape_interval: 5s
    static_configs:
    - targets: ["cadvisor:8080"]

修改配置后，Prometheus即可正常通过容器名称+内部端口访问cAdvisor，无需暴露cAdvisor到公网。

内容的提问来源于stack exchange，提问作者user2838060