Docker Compose下Driver-service无法连接Kafka Broker问题求助

架构老林

2026-8-24

Kafka容器连接问题排查与解决

问题背景

通过Docker Compose部署包含driver-service（客户端）、Kafka Broker、Zookeeper等组件的微服务集群，所有容器处于同一Docker网络中。尝试用服务名+端口（broker:29092）让driver-service连接Kafka时，出现连接失败警告：

driver-service-container | 2022-07-24 09:00:05.076 WARN 1 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

已知同一Docker网络内容器可通过服务名直接通信，但此场景下该方式无效，已参考相关技术资料配置仍未解决，需排查问题原因并给出解决方案。

相关配置文件

docker-compose.yml

version: '3'
services:

  gateway-server:
    image: gateway-server-image
    container_name: gateway-server-container
    ports:
      - '5555:5555'
    environment:
      - SECURE_KEY_USERNAME=randomSecureKeyUsername!
      - SECURE_KEY_PASSWORD=randomSecureKeyPassword!
      - PASSENGER_SERVICE_URL=172.24.2.4:4444
      - DRIVER_SERVICE_URL=172.24.2.5:3333
    networks:
      microservicesNetwork:
        ipv4_address: 172.24.2.6

  driver-service:
    image: driver-service-image
    container_name: driver-service-container
    ports:
      - '3333:3333'
    environment:
      - NOTIFICATION_SERVICE_URL=172.24.2.3:8888
      - PAYMENT_SERVICE_URL=172.24.2.2:7777
      - SECURE_KEY_USERNAME=randomSecureKeyUsername!
      - SECURE_KEY_PASSWORD=randomSecureKeyPassword!
      - KAFKA_GROUP_ID=driver-group-id
      - KAFKA_BOOTSTRAP_SERVERS=broker:29092
      - kafka.consumer.group.id=driver-group-id
      - kafka.consumer.enable.auto.commit=true
      - kafka.consumer.auto.commit.interval.ms=1000
      - kafka.consumer.auto.offset.reset=earliest
      - kafka.consumer.max.poll.records=1
    networks:
      microservicesNetwork:
        ipv4_address: 172.24.2.5

  passenger-service:
    image: passenger-service-image
    container_name: passenger-service-container
    ports:
      - '4444:4444'
    environment:
      - PAYMENT_SERVICE_URL=172.24.2.2:7777
      - SECURE_KEY_USERNAME=randomSecureKeyUsername!
      - SECURE_KEY_PASSWORD=randomSecureKeyPassword!
    networks:
      microservicesNetwork:
        ipv4_address: 172.24.2.4

  notification-service:
    image: notification-service-image
    container_name: notification-service-container
    ports:
      - '8888:8888'
    environment:
      - SECURE_KEY_USERNAME=randomSecureKeyUsername!
      - SECURE_KEY_PASSWORD=randomSecureKeyPassword!
    networks:
      microservicesNetwork:
        ipv4_address: 172.24.2.3

  payment-service:
    image: payment-service-image
    container_name: payment-service-container
    ports:
      - '7777:7777'
    environment:
      - SECURE_KEY_USERNAME=randomSecureKeyUsername!
      - SECURE_KEY_PASSWORD=randomSecureKeyPassword!
    networks:
      microservicesNetwork:
        ipv4_address: 172.24.2.2

  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - microservicesNetwork

  broker:
    image: confluentinc/cp-kafka:7.0.1
    container_name: broker
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      GROUP_ID: driver-group-id
      KAFKA_CREATE_TOPICS: "product"
    networks:
      - microservicesNetwork

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8080:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=broker
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:29092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181
      - KAFKA_CLUSTERS_0_READONLY=true
    networks:
      - microservicesNetwork

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    platform: linux/x86_64
    environment:
      - discovery.type=single-node
      - max_open_files=65536
      - max_content_length_in_bytes=100000000
      - transport.host= elasticsearch
    volumes:
      - $HOME/app:/var/app
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - microservicesNetwork

  postgresql:
    image: postgres:11.1-alpine
    platform: linux/x86_64
    container_name: postgresql
    volumes:
      - ./postgresql/:/var/lib/postgresql/data/
    environment:
      - POSTGRES_PASSWORD=123456
      - POSTGRES_USER=postgres
      - POSTGRES_DB=cqrs_db
    ports:
      - "5432:5432"
    networks:
      - microservicesNetwork

networks:
  microservicesNetwork:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.24.2.0/16
          gateway: 172.24.2.1

application.prod.properties

#datasource
spring.datasource.url=jdbc:h2:mem:db_driver
spring.datasource.username=root
spring.datasource.password=1234
spring.datasource.driver-class-name=org.h2.Driver
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
#need spring-security config.
spring.h2.console.enabled=false
spring.h2.console.path=/h2-console
spring.jpa.show-sql=true
service.security.secure-key-username=${SECURE_KEY_USERNAME}
service.security.secure-key-password=${SECURE_KEY_PASSWORD}

payment.service.url=${PAYMENT_SERVICE_URL}
notification.service.url=${NOTIFICATION_SERVICE_URL}

#kafka configs
kafka.bootstrap.servers=${KAFKA_BOOTSTRAP_SERVERS}
kafka.group.id =${KAFKA_GROUP_ID}
spring.cache.cache-names=driver
spring.jackson.serialization.fail-on-empty-beans= false
spring.http.multipart.max-file-size=10MB
spring.http.multipart.max-request-size=11MB

问题原因分析

错误日志显示driver-service实际尝试连接localhost:9092而非配置的broker:29092，核心原因在于Kafka的监听器机制：

Kafka的KAFKA_ADVERTISED_LISTENERS同时声明了PLAINTEXT://localhost:9092（宿主机外部访问）和PLAINTEXT_INTERNAL://broker:29092（Docker内部访问）。
客户端初始连接broker:29092成功后，Kafka会返回自己的advertised.listeners地址供后续通信。若客户端初始连接对应的监听器返回的是localhost:9092，客户端就会尝试访问自身容器的localhost，自然无法连接Kafka容器。
需确认driver-service是否正确使用了Kafka内部监听器对应的协议，确保Kafka返回broker:29092作为通信地址。

解决方案

步骤1：调整Kafka Broker配置

在broker服务的ports中添加内部端口映射，确保内部监听器端口可被容器访问：

ports:
  - "9092:9092"
  - "29092:29092"

环境变量保持现有配置，确保监听器映射与声明地址正确关联：

environment:
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092

步骤2：修正driver-service的Kafka配置

确保客户端配置正确加载内部监听器地址：

验证application.prod.properties中kafka.bootstrap.servers确实指向broker:29092，且环境变量KAFKA_BOOTSTRAP_SERVERS已正确注入容器。
若使用Spring Kafka，补充以下配置确保客户端行为正确：

spring.kafka.bootstrap-servers=${KAFKA_BOOTSTRAP_SERVERS}
spring.kafka.consumer.group-id=${KAFKA_GROUP_ID}
spring.kafka.properties.client.dns.lookup=use_all_dns_ips

若使用原生Kafka客户端，需确保bootstrap.servers明确设置为broker:29092。

步骤3：验证网络连通性

在driver-service容器内执行以下命令，测试与Kafka的连通性：

ping broker
telnet broker 29092

若无法连通，检查所有容器是否都加入了microservicesNetwork，以及网络子网、网关配置是否正确。

步骤4：重启服务

修改配置后重启所有相关容器：

docker-compose down
docker-compose up -d

额外排查点

在driver-service容器内执行echo $KAFKA_BOOTSTRAP_SERVERS，确认环境变量是否被正确读取。
查看Kafka容器日志，检查监听器初始化是否正常：

docker logs broker

若使用Spring Boot，确认配置项名称正确（原生kafka.bootstrap.servers与Spring的spring.kafka.bootstrap-servers为不同配置项）。

内容的提问来源于stack exchange，提问作者Semih

Docker Compose下Driver-service无法连接Kafka Broker问题求助

问题背景 #

相关配置文件 #

docker-compose.yml #

application.prod.properties #

问题原因分析 #

解决方案 #

步骤1：调整Kafka Broker配置 #

步骤2：修正driver-service的Kafka配置 #

步骤3：验证网络连通性 #

步骤4：重启服务 #

额外排查点 #