使用Docker Compose安装Scorpio Broker时的Kafka DNS解析异常问题

在Ubuntu机器上部署Scorpio Broker（依赖Kafka与PostgreSQL），使用官方Docker Compose文件启动后，Scorpio容器无法解析kafka服务的DNS，无法构建Kafka消费者，启动失败。

错误日志 #

scorpio    | 2024-12-23 12:27:46,192 WARN  [org.apa.kaf.cli.ClientUtils] (smallrye-kafka-consumer-thread-0) Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
scorpio    | 2024-12-23 12:27:46,198 ERROR [io.sma.rea.mes.provider] (main) SRMSG00230: Unable to create the publisher or subscriber during initialization: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
scorpio    |    at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.<init>(LegacyKafkaConsumer.java:264)
scorpio    |    at org.apache.kafka.clients.consumer.internals.ConsumerDelegateCreator.create(ConsumerDelegateCreator.java:65)
scorpio    |    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:600)
scorpio    |    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:595)
scorpio    |    at io.smallrye.reactive.messaging.kafka.impl.ReactiveKafkaConsumer.lambda$new$0(ReactiveKafkaConsumer.java:124)
scorpio    |    at java.base/java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:210)
scorpio    |    at io.smallrye.reactive.messaging.kafka.impl.ReactiveKafkaConsumer.lambda$new$1(ReactiveKafkaConsumer.java:120)
scorpio    |    at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
scorpio    |    at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:28)
scorpio    |    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
scorpio    |    at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
scorpio    |    at io.smallrye.mutiny.operators.uni.UniMemoizeOp.subscribe(UniMemoizeOp.java:59)
scorpio    |    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
scorpio    |    at io.smallrye.mutiny.operators.uni.UniRunSubscribeOn.lambda$subscribe$0(UniRunSubscribeOn.java:27)
scorpio    |    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
scorpio    |    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
scorpio    |    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
scorpio    |    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
scorpio    |    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
scorpio    |    at java.base/java.lang.Thread.run(Thread.java:1570)
scorpio    | Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
scorpio    |    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:103)
scorpio    |    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:62)
scorpio    |    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:58)
scorpio    |    at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.<init>(LegacyKafkaConsumer.java:183)

使用的Docker Compose文件 #

version: '3'

services:
  zookeeper:
    image: zookeeper
    ports:
      - "2181"
  kafka:
    image: bitnami/kafka:3.5.1-debian-11-r6
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_MS: 10000
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 5000
      ALLOW_PLAINTEXT_LISTENER: "yes"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
  postgres:
    image: postgis/postgis
    ports:
      - "5432"
    environment:
      POSTGRES_USER: ngb
      POSTGRES_PASSWORD: ngb
      POSTGRES_DB: ngb
  scorpio:
    image: scorpiobroker/all-in-one-runner:java-kafka-latest
    ports:
      - "9090:9090"
    environment:
      DBHOST: postgres
      BUSHOST: kafka
    depends_on:
      - postgres
      - kafka

解决方案 #

1. 修正Kafka的监听器配置 #

Bitnami Kafka镜像的环境变量与官方Kafka规范不同，原配置中的KAFKA_ADVERTISED_HOST_NAME和KAFKA_ADVERTISED_PORT需要替换为KAFKA_ADVERTISED_LISTENERS，明确指定容器间通信的地址：

kafka:
  image: bitnami/kafka:3.5.1-debian-11-r6
  ports:
    - "9092"
  environment:
    KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_LOG_RETENTION_MS: 10000
    KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 5000
    ALLOW_PLAINTEXT_LISTENER: "yes"
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  depends_on:
    - zookeeper

PLAINTEXT://kafka:9092用于Docker容器内部通信，Scorpio可以通过这个地址正常解析Kafka服务。

2. 添加服务就绪等待逻辑 #

depends_on仅保证容器启动顺序，不确保服务完全就绪。给Scorpio容器添加启动前检查，等待Kafka和PostgreSQL服务可用后再启动应用：

scorpio:
  image: scorpiobroker/all-in-one-runner:java-kafka-latest
  ports:
    - "9090:9090"
  environment:
    DBHOST: postgres
    BUSHOST: kafka
  depends_on:
    - postgres
    - kafka
  command: ["/bin/sh", "-c", "while ! nc -z kafka 9092; do sleep 1; done; while ! nc -z postgres 5432; do sleep 1; done; exec java -jar /app.jar"]

该命令会循环检查Kafka（9092端口）和PostgreSQL（5432端口）的连通性，确认就绪后启动Scorpio。

3. 显式定义Docker网络（可选） #

默认Docker Compose会创建自定义网络，但显式定义可避免潜在的网络解析问题：

version: '3'

networks:
  scorpio-network:
    driver: bridge

services:
  zookeeper:
    image: zookeeper
    ports:
      - "2181"
    networks:
      - scorpio-network
  kafka:
    # 保持修正后的配置
    networks:
      - scorpio-network
  postgres:
    # 保持原配置
    networks:
      - scorpio-network
  scorpio:
    # 保持修正后的配置
    networks:
      - scorpio-network

内容的提问来源于stack exchange，提问作者Saikiran Vathipalli