Docker容器内NestJS连接Kafka微服务失败排查方案

我有一个运行在Docker中的NestJS应用，计划将部分功能迁移至微服务，选用Kafka作为消息中间件。但微服务始终无法连接到Kafka容器，以下是相关配置与报错信息：

Docker Compose配置 #

a-microservice:
    build:
      context: ../back-a-microservice
      dockerfile: ../back-a-microservice/Dockerfile
    container_name: a-microservice
    env_file: ../back/.env
    networks:
      - app-network
    depends_on:
      - postgres
      - back
      - kafka1
    restart: always
    command: npm run start:migrate:dev

kafka1:
    image: bitnami/kafka:latest
    ports:
      - 9092:9092
      - 9094:9094
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka1:9092,EXTERNAL://kafka1:9094
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka1:9093
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_CFG_NUM_PARTITIONS=2
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - volume-kafka-1:/bitnami/kafka
    depends_on:
      - postgres
      - back
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

volumes:
  postgres:
    driver: local
  volume-kafka-1:
    driver: local

NestJS初始化代码 #

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(
    AppModule,
    {
      transport: Transport.KAFKA,
      options: {
        client: {
          brokers: ['localhost:9092'],
        },
        consumer: {
          groupId: 'a-consumer',
        },
      },
    },
  );
  await app.listen();
}
bootstrap();

依赖配置（package.json） #

"dependencies": {
    "@nestjs/common": "^10.0.0",
    "@nestjs/core": "^10.0.0",
    "@nestjs/microservices": "^10.3.8",
    "kafkajs": "^2.2.4"
  },

报错信息 #

2024-04-24 19:04:35 [Nest] 93  - 04/24/2024, 4:04:35 PM   ERROR [ServerKafka] ERROR [Connection] Connection error:  {"timestamp":"2024-04-24T16:04:35.795Z","logger":"kafkajs","broker":"localhost:9092","clientId":"nestjs-consumer-server","stack":"AggregateError: \n    at internalConnectMultiple (node:net:1116:18)\n    at afterConnectMultiple (node:net:1683:7)"}
2024-04-24 19:04:35 [Nest] 93  - 04/24/2024, 4:04:35 PM   ERROR [ServerKafka] ERROR [BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error:  {"timestamp":"2024-04-24T16:04:35.797Z","logger":"kafkajs","retryCount":0,"retryTime":248}

已尝试更换端口、将brokers改为docker-compose中的kafka1主机名、单独重启微服务容器，但均未解决问题。Kafka容器运行正常，日志显示已在0.0.0.0:9092和0.0.0.0:9094等待连接。

1. 修正NestJS中的Kafka Broker地址 #

Docker容器内的localhost指向容器自身，而非宿主机或其他容器。两个容器同属app-network网络，直接使用Kafka容器的服务名kafka1作为主机名，匹配Kafka内部PLAINTEXT监听器的端口9092：

修改NestJS初始化代码：

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(
    AppModule,
    {
      transport: Transport.KAFKA,
      options: {
        client: {
          brokers: ['kafka1:9092'],
        },
        consumer: {
          groupId: 'a-consumer',
        },
      },
    },
  );
  await app.listen();
}
bootstrap();

2. 解决容器启动顺序问题 #

depends_on仅保证容器启动顺序，不保证Kafka服务完全就绪。开发环境下可在微服务启动前添加延迟，或使用工具等待Kafka端口就绪：

修改Docker Compose中a-microservice的command：

command: >
  sh -c "sleep 10 && npm run start:migrate:dev"

3. 验证网络连通性 #

进入微服务容器，测试与Kafka的连通性：

# 进入容器
docker exec -it a-microservice bash

# 测试Kafka主机可达性
ping kafka1

# 测试端口连通性
telnet kafka1 9092

如果连通失败，执行docker network inspect app-network确认两个容器都加入了app-network网络。

4. 确认Kafka服务状态 #

查看Kafka容器日志，确认服务完全启动：

docker logs kafka1

寻找[KafkaServer id=1] started类日志，表明Kafka已就绪。

内容的提问来源于stack exchange，提问作者SnukSmyk