Grafana Loki通信503错误：无法获取标签问题求助

此前用docker-compose部署Grafana、Loki、Promtail、Kafka栈完全正常，但切换到Ubuntu环境通过Jenkins+Ansible用Podman单独部署后出现以下问题：

• Kafka与Promtail通信正常，Promtail可向Loki推送日志
• Grafana配置Loki数据源时提示：Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details
• 宿主机访问http://IP_ADDRESS:3100/ready返回ready，但访问/loki/api/v1/labels接口超时无响应
• Kafka容器内执行curl http://localhost:3100/ready返回503 Service Unavailable

原正常运行的docker-compose.yml #

version: "3.7"

networks:
  kafka-net:
    name: kafka-net
    driver: bridge

services:
  kafka_broker:
    image: 'bitnami/kafka:latest'
    container_name: kafka_broker
    networks:
      - kafka-net
    ports:
      - '9094:9094'
    environment:
        KAFKA_BROKER_ID: 1
        KAFKA_ENABLE_KRAFT: 'yes'
        KAFKA_CFG_PROCESS_ROLES: broker,controller
        KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
        KAFKA_CFG_LISTENERS: CLIENT://:29092,CONTROLLER://:9093,EXTERNAL://:9094
        KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
        KAFKA_CFG_INTER_BROKER_LISTENER_NAME: CLIENT
        KAFKA_CFG_ADVERTISED_LISTENERS: CLIENT://kafka_broker:29092,EXTERNAL://localhost:9094
        KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
        KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
        KAFKA_DEFAULT_REPLICATION_FACTOR: 1
        KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        ALLOW_PLAINTEXT_LISTENER: 'yes'

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    networks:
      - kafka-net
    ports:
      - '8080:8080'
    environment:
      KAFKA_CLUSTERS_0_NAME: "local"
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka_broker:29092"
    depends_on:
      - "kafka_broker"

  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - kafka-net

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /var/workspace/promtail/docker-config.yml:/etc/promtail/docker-config.yml
    command: -config.file=/etc/promtail/docker-config.yml
    networks:
      - kafka-net

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    networks:
      - kafka-net

Podman部署相关配置 #

Loki配置文件 #

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

analytics:
  reporting_enabled: false

Promtail配置文件 #

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: varlogs
  static_configs:
    - targets: [localhost]
      labels:
        job: varlogs
        __path__: /var/logtest/system.log
- job_name: kafka
  kafka:
    brokers:
    - kafka_broker:29092
    topics:
    - grafana
    labels:
      job: kafka
      host: ${HOSTNAME:promtail}
  relabel_configs:
      - action: replace
        source_labels:
          - __meta_kafka_topic
        target_label: topic
      - action: replace
        source_labels:
          - __meta_kafka_partition
        target_label: partition
      - action: replace
        source_labels:
          - __meta_kafka_group_id
        target_label: group

Kafka环境变量文件kafka-env.env #

KAFKA_BROKER_ID=1
KAFKA_ENABLE_KRAFT=yes
KAFKA_CFG_PROCESS_ROLES=broker,controller
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
KAFKA_CFG_LISTENERS=CLIENT://:29092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka_broker:29092,EXTERNAL://localhost:9094
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093
KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
KAFKA_DEFAULT_REPLICATION_FACTOR=1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
ALLOW_PLAINTEXT_LISTENER=yes

Podman启动命令 #

podman run -d --env-file=/kafka/env/kafka-env.env -v /kafka/data:/bitnami/kafka --name kafka_broker --net=mycustom-bridge-net bitnami/kafka:latest

podman run -d -p 3100:3100 -v /grafana-loki/data:/loki -v /grafana-loki/config/local-config.yml:/etc/loki/local-config.yaml --name loki --net=mycustom-bridge-net grafana/loki:latest -config.file=/etc/loki/local-config.yaml

podman run -d -v /var/lib/containers:/var/lib/containers -v /grafana-promtail/config/docker-config.yml:/etc/promtail/docker-config.yml --name promtail --net=mycustom-bridge-net grafana/promtail:latest -config.file=/etc/promtail/docker-config.yml

podman run -d -p 3000:3000 -v /grafana/data:/var/lib/grafana --name grafana --net=mycustom-bridge-net grafana/grafana:latest

问题排查与解决思路 #

1. 容器间网络访问错误

   • Podman自定义网桥内，容器需通过容器名访问而非localhost：
     • 修改Promtail配置中的Loki地址为http://loki:3100/loki/api/v1/push
     • 容器内访问Loki需使用http://loki:3100而非localhost

2. Loki网络绑定配置缺陷

   • Loki默认仅绑定127.0.0.1，导致容器外/其他容器无法访问完整API，修改Loki配置：

     server:
       http_listen_address: 0.0.0.0
       http_listen_port: 3100
       grpc_listen_port: 9096
     

   • 调整ring.instance_addr为0.0.0.0或删除该配置，让Loki自动识别容器IP，避免ring组件初始化异常导致API超时

3. 存储目录权限与路径不匹配

   • Loki配置的path_prefix为/tmp/loki，但挂载了宿主机/grafana-loki/data:/loki，修改Loki配置的path_prefix为/loki
   • 确保宿主机目录权限匹配Loki容器默认用户：chown -R 10001:10001 /grafana-loki/data

4. 自定义网桥验证

   • 检查网桥配置：podman network inspect mycustom-bridge-net
   • 验证容器间连通性：podman exec kafka_broker ping -c 3 loki、podman exec grafana ping -c 3 loki

5. 容器启动顺序优化

   • 添加启动依赖，或用脚本检查Loki的/ready接口返回正常后，再启动Promtail、Grafana等服务

内容的提问来源于stack exchange，提问作者Peter S.