Docker环境下Apache Pinot连接Kafka元数据获取超时问题求助

在Docker环境中尝试执行bash bin/pinot-admin.sh AddTable命令给Apache Pinot添加表时，触发Timeout expired while fetching topic metadata错误，具体日志和Docker Compose配置如下：

命令执行日志 #

sh-5.2# bash bin/pinot-admin.sh AddTable -tableConfigFile /tmp/pinot/tb.json -schemaFile /tmp/pinot/sch.json -exec
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:gc-pinot-controller.log instead.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024/04/04 19:04:09.710 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /tmp/pinot/tb.json -offlineTableConfigFile null -realtimeTableConfigFile null -schemaFile /tmp/pinot/sch.json -controllerProtocol http -controllerHost 172.18.0.4 -controllerPort 9000 -user null -password [hidden] -exec
2024/04/04 19:04:42.382 INFO [AddTableCommand] [main] {"code":500,"error":"org.apache.pinot.spi.stream.TransientConsumerException: org.apache.pinot.shaded.org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata"}

Docker Compose配置 #

version: '3.7'
services:
 
  zookeeper:
      image: 'bitnami/zookeeper:latest'
      ports:
        - '2181:2181'
      environment:
        - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper


  pinot-controller:
    image: apachepinot/pinot:latest
    hostname: pinot-controller
    ports:
      - "9000:9000"
    command: StartController -zkAddress zookeeper:2181
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:gc-pinot-controller.log"
    depends_on:
      - kafka

  pinot-broker:
    image: apachepinot/pinot:latest
    hostname: pinot-broker
    ports:
      - "8099:8099"
    command: StartBroker -zkAddress zookeeper:2181
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:gc-pinot-broker.log"
    depends_on:
      - zookeeper
      - kafka
      - pinot-controller

  pinot-server:
    image: apachepinot/pinot:latest
    hostname: pinot-server
    ports:
      - "8098:8098"
    command: StartServer -zkAddress zookeeper:2181
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:gc-pinot-server.log"
    depends_on:
      - zookeeper
      - kafka
      - pinot-controller

错误原因分析 #

• Kafka地址配置错误：Docker Compose中Kafka的KAFKA_CFG_ADVERTISED_LISTENERS设置为PLAINTEXT://127.0.0.1:9092，但Pinot服务运行在Docker容器内部，容器内的127.0.0.1指向容器自身，而非宿主机的Kafka服务。Pinot尝试连接127.0.0.1:9092时无法访问到实际的Kafka broker，导致元数据获取超时。
• 服务就绪状态未验证：Docker的depends_on仅保证服务启动顺序，不确保服务完全就绪。Kafka可能还未完成初始化，Pinot就尝试连接，也会触发超时。

解决方法 #

1. 修正Kafka的Advertised Listeners配置：将Kafka的KAFKA_CFG_ADVERTISED_LISTENERS改为容器可访问的服务名地址：

   KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
   

   这样Docker容器内的Pinot服务可以通过服务名kafka解析到Kafka容器的IP，正常建立连接。
2. 添加Kafka健康检查：在Kafka服务中添加健康检查，确保Kafka完全启动后再启动Pinot服务：

   kafka:
     # 其他配置不变
     healthcheck:
       test: ["CMD", "kafka-topics.sh", "--list", "--bootstrap-server", "localhost:9092"]
       interval: 10s
       timeout: 5s
       retries: 5
   

   同时修改Pinot服务的depends_on，依赖Kafka的健康状态：

   pinot-controller:
     depends_on:
       kafka:
         condition: service_healthy
   

3. 验证表配置中的Kafka地址：确保Pinot表配置文件（tb.json）中指定的Kafka broker地址与修正后的advertised.listeners一致，即kafka:9092。

内容的提问来源于stack exchange，提问作者Ingrid Iplinsky