如何通过Docker Compose正确部署Kafka KRaft模式并解决报错

架构老林

2026-6-20

Kafka KRaft模式Docker Compose部署报错排查与解决方案

问题描述

尝试通过Docker Compose运行Kafka KRaft模式时持续报错，当前使用的docker-compose.yml配置如下：

services:
  # Kafka controller: manages the overall state of the Kafka cluster
  kafka-controller-0:
    image: apache/kafka:3.7.1
    container_name: kafka-controller-0
    environment:
      KAFKA_BROKER_ID: 0
      # Listener for the controller and broker
      KAFKA_CONTROLLER_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9090
      # Port for Raft communication (consensus)
      KAFKA_RAFT_PORT: 9091
      # Directory for Raft and Kafka data
      KAFKA_RAFT_DATA_DIR: /var/lib/kafka/data
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - ./data/kafka-controller-0:/var/lib/kafka/data
    ports:
      # expose broker port
      - "9092:9092"
    networks:
      - kafka-net

  # Kafka broker: stores and manages partitions of topics
  kafka-broker-0:
    image: apache/kafka:3.7.1
    container_name: kafka-broker-0
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_CONTROLLER_LISTENERS: PLAINTEXT://:9092
      KAFKA_RAFT_PORT: 9091
      KAFKA_RAFT_DATA_DIR: /var/lib/kafka/data
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - ./data/kafka-broker-0:/var/lib/kafka/data
    ports:
      # expose broker port
      - "9093:9092"
    networks:
      - kafka-net

  kafka-broker-1:
    image: apache/kafka:3.7.1
    container_name: kafka-broker-1
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_CONTROLLER_LISTENERS: PLAINTEXT://:9092
      KAFKA_RAFT_PORT: 9091
      KAFKA_RAFT_DATA_DIR: /var/lib/kafka/data
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - ./data/kafka-broker-1:/var/lib/kafka/data
    ports:
      - "9094:9092"
    networks:
      - kafka-net

networks:
  kafka-net:

报错信息

首次运行出现如下错误：

Attaching to kafka-broker-0, kafka-broker-1, kafka-controller-0
kafka-controller-0  | ===> User
kafka-broker-0      | ===> User
kafka-broker-0      | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka-controller-0  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka-broker-0      | ===> Setting default values of environment variables if not already set.
kafka-controller-0  | ===> Setting default values of environment variables if not already set.
kafka-controller-0  | CLUSTER_ID not set. Setting it to default value: "5L6g3nShT-eMCtK--X86sw"
kafka-broker-0      | CLUSTER_ID not set. Setting it to default value: "5L6g3nShT-eMCtK--X86sw"
kafka-controller-0  | ===> Configuring ...
kafka-broker-0      | ===> Configuring ...
kafka-controller-0  | ===> Launching ... 
kafka-broker-0      | ===> Launching ... 
kafka-broker-0      | ===> Using provided cluster id 5L6g3nShT-eMCtK--X86sw ...
kafka-controller-0  | ===> Using provided cluster id 5L6g3nShT-eMCtK--X86sw ...
kafka-broker-1      | ===> User
kafka-broker-1      | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka-broker-1      | ===> Setting default values of environment variables if not already set.
kafka-broker-1      | CLUSTER_ID not set. Setting it to default value: "5L6g3nShT-eMCtK--X86sw"
kafka-broker-1      | ===> Configuring ...
kafka-broker-1      | ===> Launching ... 
kafka-broker-1      | ===> Using provided cluster id 5L6g3nShT-eMCtK--X86sw ...
kafka-broker-0      | Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration `zookeeper.connect` which has no default value. at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2299) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2290) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1638) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:71) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:71) at kafka.tools.StorageTool$.main(StorageTool.scala:52) at kafka.docker.KafkaDockerWrapper$.main(KafkaDockerWrapper.scala:47) at kafka.docker.KafkaDockerWrapper.main(KafkaDockerWrapper.scala)
kafka-controller-0  | Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration `zookeeper.connect` which has no default value. at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2299) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2290) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1638) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:71) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:71) at kafka.tools.StorageTool$.main(StorageTool.scala:52) at kafka.docker.KafkaDockerWrapper$.main(KafkaDockerWrapper.scala:47) at kafka.docker.KafkaDockerWrapper.main(KafkaDockerWrapper.scala)
kafka-broker-1      | Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration `zookeeper.connect` which has no default value. at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2299) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2290) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1638) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:71) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:71) at kafka.tools.StorageTool$.main(StorageTool.scala:52) at kafka.docker.KafkaDockerWrapper$.main(KafkaDockerWrapper.scala:47) at kafka.docker.KafkaDockerWrapper.main(KafkaDockerWrapper.scala)
kafka-controller-0 exited with code 1
kafka-broker-1 exited with code 1
kafka-broker-0 exited with code 1

添加KAFKA_ZOOKEEPER_CONNECT: ""后，新错误出现：

kafka-broker-0      | The kafka configuration file appears to be for a legacy cluster. Formatting is only supported for clusters in KRaft mode.
kafka-broker-1      | The kafka configuration file appears to be for a legacy cluster. Formatting is only supported for clusters in KRaft mode.
kafka-controller-0  | The kafka configuration file appears to be for a legacy cluster. Formatting is only supported for clusters in KRaft mode.

询问正确部署方式及优质文档。

解决方案

核心问题分析

原配置缺少KRaft模式的核心标识参数，导致容器默认以ZooKeeper模式启动，触发报错。必须明确指定节点角色（控制器/ Broker），并配置KRaft模式的关键参数。

正确的Docker Compose配置

以下是可运行的KRaft模式多节点配置（1个控制器+2个Broker）：

services:
  kafka-controller-0:
    image: apache/kafka:3.7.1
    container_name: kafka-controller-0
    environment:
      # 指定节点为控制器角色
      KAFKA_NODE_ROLES: controller
      KAFKA_BROKER_ID: 0
      # 集群ID，所有节点必须一致，可通过kafka-storage.sh random-uuid生成
      CLUSTER_ID: "5L6g3nShT-eMCtK--X86sw"
      # 控制器集群投票节点配置
      KAFKA_CONTROLLER_QUORUM_VOTERS: "0@kafka-controller-0:9090"
      # 监听地址配置
      KAFKA_LISTENERS: CONTROLLER://:9090,PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-controller-0:9092
      # 监听协议映射
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      # Raft数据存储目录
      KAFKA_RAFT_DATA_DIR: /var/lib/kafka/data
    volumes:
      - ./data/kafka-controller-0:/var/lib/kafka/data
    networks:
      - kafka-net

  kafka-broker-0:
    image: apache/kafka:3.7.1
    container_name: kafka-broker-0
    environment:
      # 指定节点为Broker角色
      KAFKA_NODE_ROLES: broker
      KAFKA_BROKER_ID: 1
      CLUSTER_ID: "5L6g3nShT-eMCtK--X86sw"
      # 指定控制器集群地址
      KAFKA_CONTROLLER_QUORUM_VOTERS: "0@kafka-controller-0:9090"
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-broker-0:9092
      # 日志存储目录
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - ./data/kafka-broker-0:/var/lib/kafka/data
    ports:
      - "9093:9092"
    networks:
      - kafka-net
    depends_on:
      - kafka-controller-0

  kafka-broker-1:
    image: apache/kafka:3.7.1
    container_name: kafka-broker-1
    environment:
      KAFKA_NODE_ROLES: broker
      KAFKA_BROKER_ID: 2
      CLUSTER_ID: "5L6g3nShT-eMCtK--X86sw"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "0@kafka-controller-0:9090"
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-broker-1:9092
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - ./data/kafka-broker-1:/var/lib/kafka/data
    ports:
      - "9094:9092"
    networks:
      - kafka-net
    depends_on:
      - kafka-controller-0

networks:
  kafka-net:
    driver: bridge

关键配置说明

KAFKA_NODE_ROLES：明确节点角色为controller或broker，这是KRaft模式的核心标识，缺失则默认走ZooKeeper模式。
CLUSTER_ID：所有节点必须使用相同的集群ID，确保属于同一集群，可通过kafka-storage.sh random-uuid命令生成唯一ID。
KAFKA_CONTROLLER_QUORUM_VOTERS：指定控制器集群的投票节点，格式为brokerId@hostname:port，单控制器场景下配置为当前控制器节点信息即可。
KAFKA_LISTENERS与KAFKA_ADVERTISED_LISTENERS：配置节点的监听地址和对外公告地址，容器内部通过服务名通信，外部通过端口映射访问。

部署步骤

创建数据目录并设置权限（避免容器内用户无读写权限）：

mkdir -p ./data/kafka-controller-0 ./data/kafka-broker-0 ./data/kafka-broker-1
chown -R 1000:1000 ./data

启动容器：

docker-compose up -d

验证集群状态：

# 进入控制器容器执行命令查看集群主题
docker exec -it kafka-controller-0 kafka-topics.sh --list --bootstrap-server kafka-broker-0:9092

优质文档推荐

Apache Kafka官方KRaft部署指南：涵盖KRaft模式核心概念、配置参数、集群搭建全流程，是最权威的参考资料。
Apache Kafka Docker镜像官方文档：详细说明镜像的环境变量配置、启动逻辑，针对KRaft模式提供了专属配置示例。

内容的提问来源于stack exchange，提问作者Peter

如何通过Docker Compose正确部署Kafka KRaft模式并解决报错

问题描述 #

报错信息 #

解决方案 #

核心问题分析 #

正确的Docker Compose配置 #

关键配置说明 #

部署步骤 #

优质文档推荐 #