MacOS下Docker部署Spring Boot+Kafka微服务连接失败求助

我在MacOS上有一个基于PostgreSQL和Kafka的Java Spring Boot微服务应用，包含filmservice（Kafka生产者）和eventservice（Kafka消费者）两个模块。当在Docker容器中启动时，应用模块无法连接到Kafka服务器，但本地IDE启动时能正常连接容器内的Kafka。

报错信息 #

filmservice（生产者）报错： #

2024-09-05 14:12:10 filminator-filmservice  | 2024-09-05T11:12:10.978Z  INFO 1 --- [           main] o.a.k.c.c.internals.LegacyKafkaConsumer  : [Consumer clientId=consumer-my-group-id-1, groupId=my-group-id] Subscribed to topic(s): my-topic
2024-09-05 14:12:10 filminator-filmservice  | 2024-09-05T11:12:10.990Z  INFO 1 --- [           main] c.z.f.filmservice.FilminatorFilmService  : Started FilminatorFilmService in 1.974 seconds (process running for 2.223)
2024-09-05 14:12:11 filminator-filmservice  | 2024-09-05T11:12:11.102Z  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-my-group-id-1, groupId=my-group-id] Node -1 disconnected.
2024-09-05 14:12:11 filminator-filmservice  | 2024-09-05T11:12:11.103Z  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-my-group-id-1, groupId=my-group-id] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.
2024-09-05 14:12:11 filminator-filmservice  | 2024-09-05T11:12:11.103Z  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-my-group-id-1, groupId=my-group-id] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

eventservice（消费者）报错： #

2024-09-05 14:14:58 filminator-eventservice  | 2024-09-05T11:14:58.843Z  WARN 1 --- [ntainer#1-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-my-group-id-2, groupId=my-group-id] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.

相关配置 #

docker-compose.yml #

services:

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2182:2182'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

  userservice:
    build: eventservice
    image: eventservice-image
    container_name: filminator-eventservice
    ports:
      - "8082:8082"
    depends_on:
      - kafka
      - filmservice
      - postgres_db
    environment:
      - SPRING_DATASOURCE_URL=jdbc:postgresql://postgres_db:5432/filminator

  filmservice:
      build: filmservice
      image: filmservice-image
      container_name: filminator-filmservice
      ports:
        - "8081:8081"
      depends_on:
        - kafka
        - postgres_db
      environment:
        - SPRING_DATASOURCE_URL=jdbc:postgresql://postgres_db:5432/filminator


  postgres_db:
      image: postgres:13.7-alpine
      volumes:
        - /var/lib/postgresql/data/
      container_name: db_postgres
      ports:
        - "6541:5432"
      environment:
        - POSTGRES_DB=filminator
        - POSTGRES_USER=postgres
        - POSTGRES_PASSWORD=iamroot

Dockerfile #

FROM openjdk:19
COPY target/*.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

application.properties #

filmservice

server.port=8081
spring.datasource.driver-class-name=org.postgresql.Driver
spring.sql.init.mode=always
spring.datasource.url=jdbc:postgresql://localhost:5432/filminator
spring.datasource.username=postgres
spring.datasource.password=iamroot

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group-id

eventservice

server.port=8082
# 其他配置与filmservice一致

疑问 #

1. 是否可以将应用和Kafka服务器部署在同一个Docker容器中？
2. 如何解决当前的连接问题？

关于是否可以将应用与Kafka部署在同一个容器 #

不建议这么做。Docker的设计核心是一个容器运行一个进程，混部署会带来以下问题：

• 任一进程崩溃会导致整个容器重启，牵连另一服务；
• 无法单独对Kafka或应用进行水平扩展；
• 日志、资源隔离混乱，排查问题难度大幅提升。

解决当前Kafka连接问题 #

问题根源是：容器内的应用用localhost:9092连接Kafka，但容器的localhost指向自身，而非Kafka容器。需从以下几点调整：

1. 修改Kafka的监听配置 #

在docker-compose的kafka服务中，配置双监听端口，同时支持本地IDE和容器内应用访问：

kafka:
  image: 'bitnami/kafka:latest'
  ports:
    - '9092:9092'
    - '9093:9093' # 新增容器内部通信端口
  environment:
    - KAFKA_BROKER_ID=1
    - KAFKA_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_INTERNAL://:9093
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092,PLAINTEXT_INTERNAL://kafka:9093
    - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
    - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT_INTERNAL
    - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
    - ALLOW_PLAINTEXT_LISTENER=yes

配置后：

• 本地IDE通过127.0.0.1:9092访问Kafka；
• 容器内应用通过kafka:9093访问Kafka。

2. 覆盖Spring Boot应用的Kafka配置 #

在docker-compose的filmservice和userservice（即eventservice）中添加环境变量，替换容器内的Kafka地址：

filmservice:
  # ... 原有配置
  environment:
    - SPRING_DATASOURCE_URL=jdbc:postgresql://postgres_db:5432/filminator
    - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9093 # 新增

userservice:
  # ... 原有配置
  environment:
    - SPRING_DATASOURCE_URL=jdbc:postgresql://postgres_db:5432/filminator
    - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9093 # 新增

本地IDE使用的application.properties保持spring.kafka.bootstrap-servers=localhost:9092不变，实现本地和容器环境兼容。

3. 补充服务就绪检查 #

depends_on仅保证容器启动顺序，不保证服务完全就绪。添加健康检查确保Kafka和PostgreSQL启动完成后再启动应用：

kafka:
  # ... 原有配置
  healthcheck:
    test: ["CMD", "kafka-topics.sh", "--list", "--bootstrap-server", "localhost:9092"]
    interval: 10s
    timeout: 5s
    retries: 5

filmservice:
  # ... 原有配置
  depends_on:
    kafka:
      condition: service_healthy
    postgres_db:
      condition: service_healthy

userservice:
  # ... 原有配置
  depends_on:
    kafka:
      condition: service_healthy
    filmservice:
      condition: service_started
    postgres_db:
      condition: service_healthy

postgres_db:
  # ... 原有配置
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U postgres -d filminator"]
    interval: 10s
    timeout: 5s
    retries: 5

内容的提问来源于stack exchange，提问作者Zholtikov Alexander