Docker内微服务无法连接同环境Kafka问题求助

我有一个docker-compose.yml文件，用来定义通过Kafka通信的微服务（driver-service、ride-service）以及Kafka集群本身。启动docker-compose时遇到Kafka连接错误，但单独在Docker中启动Kafka、本地运行微服务时能正常连接。

1. docker-compose服务配置 #

driver-service配置 #

driver_service:
    image: driver-service
    container_name: driver-service
    environment:
      - SPRING_DATASOURCE_URL=jdbc:mysql://mysql_db:3306/drivers
      - SPRING_DATASOURCE_USERNAME=root
      - SPRING_DATASOURCE_PASSWORD=root
      - EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://eureka-service:8090/eureka
      - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:29092
      - KAFKA_BOOTSTRAP_SERVERS=kafka:29092
    ports:
      - '8084:8084'
    networks:
      - cab_aggregator
    depends_on:
      - eureka-service
      - mysql_db
      - kafka

ride-service配置 #

ride_service:
    image: ride-service
    container_name: ride-service
    environment:
      - SPRING_DATASOURCE_URL=jdbc:mysql://mysql_db:3306/rides
      - SPRING_DATASOURCE_USERNAME=root
      - SPRING_DATASOURCE_PASSWORD=root
      - EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE=http://eureka-service:8090/eureka
      - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:29092
      - KAFKA_BOOTSTRAP_SERVERS=kafka:29092
    ports:
      - '8082:8082'
    networks:
      - cab_aggregator
    depends_on:
      - eureka-service
      - mysql_db
      - kafka

Kafka与Zookeeper配置 #

zookeeper:
    image: zookeeper:3.7.0
    container_name: zookeeper
    networks:
      - cab_aggregator
    restart: "no"
    ports:
      - "2181:2181"

kafka:
    image: obsidiandynamics/kafka
    container_name: kafka
    restart: "no"
    networks:
      - cab_aggregator
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER_INTERNAL:PLAINTEXT,DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_LISTENERS: DOCKER_INTERNAL://:29092,DOCKER_EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: DOCKER_INTERNAL://kafka:29092,DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper

kafdrop:
    image: obsidiandynamics/kafdrop
    container_name: kafdrop
    restart: "no"
    networks:
      - cab_aggregator
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka:29092"
    depends_on:
      - "kafka"

2. 启动时的错误日志 #

rg.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater [WARN ] 19:10:19 - [Consumer clientId=consumer-ride-creation-group-1, groupId=ride-creation-group] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
2024-02-17T19:10:20.622588304Z org.apache.kafka.clients.NetworkClient [INFO ] 19:10:20 - [Consumer clientId=consumer-ride-creation-group-1, groupId=ride-creation-group] Node -1 disconnected.
2024-02-17T19:10:20.622669266Z org.apache.kafka.clients.NetworkClient [WARN ] 19:10:20 - [Consumer clientId=consumer-ride-creation-group-1, groupId=ride-creation-group] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-02-17T19:10:20.622859991Z org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater [WARN ] 19:10:20 - [Consumer clientId=consumer-ride-creation-group-1, groupId=ride-creation-group] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

3. 微服务application配置 #

spring:
  datasource:
    url: jdbc:mysql://localhost:3306/rides
    username: ${DB_USERNAME:root}
    password: ${DB_PASSWORD:root}
    driver-class-name: com.mysql.cj.jdbc.Driver
  jpa:
    properties:
      hibernate:
        dialect: org.hibernate.dialect.MySQLDialect
    show-sql: true
    hibernate:
      ddl-auto: none
  liquibase:
    change-log: classpath:/db/changelog/db.changelog-master.xml
  application:
    name: ride
  kafka:
    consumer:
      group-id: driver-creation-group
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      bootstrap-servers: localhost:9092
topic:
  name:
    ride: ride
    driver: driver
    available: available
service:
    passenger:
      name: passenger
      path: /api/v1/passengers
    driver:
      name: driver
      path: /api/v1/drivers
    payment:
      name: payment
      path: /api/v1/payment
eureka:
  client:
    service-url:
      defaultZone: http://localhost:8090/eureka

logging:
  pattern:
    console: "%C{1.} [%-5level] %d{HH:mm:ss} - %msg%n"
server:
  port: 8082

问题分析与解决 #

从错误日志能明确看到：微服务容器内的Kafka客户端在尝试连接localhost:9092，但Docker容器的localhost指向容器自身，而非Kafka容器或宿主机，自然连不上。

虽然docker-compose里给微服务配置了SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:29092环境变量，但微服务的application.yml里硬编码了bootstrap-servers: localhost:9092，如果配置文件的优先级高于环境变量，就会导致客户端忽略环境变量，继续用localhost连接。

解决步骤： #

1. 修改微服务application.yml，将Kafka的bootstrap-servers改为读取环境变量，保留本地运行的默认值：

spring:
  kafka:
    consumer:
      bootstrap-servers: ${SPRING_KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
    producer:
      bootstrap-servers: ${SPRING_KAFKA_BOOTSTRAP_SERVERS:localhost:9092}

这样本地运行时使用localhost:9092，容器内运行时自动读取docker-compose配置的kafka:29092。

2. 重新构建微服务镜像，确保修改后的配置被打包进driver-service和ride-service镜像中。

3. 验证Docker网络，确认所有服务都在cab_aggregator网络内，Docker会自动解析容器名称kafka为对应的IP地址，保证微服务能访问到Kafka容器。

4. 确认配置优先级：Spring Boot中环境变量的优先级高于application.yml，修改后环境变量会覆盖配置文件的默认值，确保容器内使用正确的Kafka地址。

内容的提问来源于stack exchange，提问作者Назар