Docker Compose部署Kafka遇配置缺失等问题求助

问题概述 #

• 原教程指定的bitnami/kafka:3.6镜像无法拉取，报错：

  kafka Error {"message":"manifest unknown: manifest unknown"}
  

• 替换为apache/kafka:latest后，容器启动但日志提示缺少process.roles配置（已设置KAFKA_CFG_PROCESS_ROLES仍无效），错误日志：

  uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
  ===> Setting default values of environment variables if not already set.
  CLUSTER_ID not set. Setting it to default value: "5L6g3nShT-eMCtK--X86sw"
  ===> Configuring ...
  ===> Launching ... 
  ===> Using provided cluster id 5L6g3nShT-eMCtK--X86sw ...
  Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "process.roles" which has no default value.
  	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:534)
  	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524)
  	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)
  	at kafka.server.config.AbstractKafkaConfig.<init>(AbstractKafkaConfig.java:72)
  	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:156)
  	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:158)
  	at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:79)
  	at scala.Option.flatMap(Option.scala:283)
  	at kafka.tools.StorageTool$.execute(StorageTool.scala:79)
  	at kafka.tools.StorageTool$.main(StorageTool.scala:46)
  	at kafka.docker.KafkaDockerWrapper$.main(KafkaDockerWrapper.scala:57)
  	at kafka.docker.KafkaDockerWrapper.main(KafkaDockerWrapper.scala)
  

• 替换为apache/kafka:3.7.0后，报错缺少zookeeper.connect配置，错误日志：

  uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
  ===> Setting default values of environment variables if not already set.
  CLUSTER_ID not set. Setting it to default value: "5L6g3nShT-eMCtK--X86sw"
  ===> Configuring ...
  ===> Launching ... 
  ===> Using provided cluster id 5L6g3nShT-eMCtK--X86sw ...
  Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration `zookeeper.connect` which has no default value.
  	at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2299)
  	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2290)
  	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1638)
  	at kafka.tools.StorageTool$.$anonfun$main$1(StorageTool.scala:52)
  	at scala.Option.flatMap(Option.scala:283)
  	at kafka.tools.StorageTool$.main(StorageTool.scala:52)
  	at kafka.docker.KafkaDockerWrapper$.main(KafkaDockerWrapper.scala:47)
  	at kafka.docker.KafkaDockerWrapper.main(KafkaDockerWrapper.scala)
  

当前使用的docker-compose.yml内容：

services:
  mariadb:
    container_name: mariadb
    image: 'mariadb:11.4.2'
    environment:
      - MYSQL_ROOT_PASSWORD=***************
      - MYSQL_USER=spark
      - MYSQL_PASSWORD=***************
      - MYSQL_DATABASE=spark_de
    ports:
      - '3306:3306'
  kafka:
    image: 'apache/kafka:latest'
    container_name: kafka-broker
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_BROKER_ID=1001
      - KAFKA_CFG_NODE_ID=1001
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - KAFKA_CFG_LISTENERS=CLIENT://:29092,EXTERNAL://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:29092,EXTERNAL://localhost:9092
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1001@127.0.0.1:9093
    ports:
      - '9092:9092'
  redis:
    container_name: redis
    image: 'redis:7.2.5-alpine'
    ports:
      - '6379:6379'

问题原因 #

1. 环境变量前缀不匹配：apache/kafka镜像的环境变量不需要KAFKA_CFG_前缀，直接使用Kafka配置项的全大写形式（点号替换为下划线），之前设置的KAFKA_CFG_PROCESS_ROLES未被容器识别。
2. Kraft模式配置不完整：CONTROLLER_QUORUM_VOTERS使用127.0.0.1会指向容器自身，无法正确访问控制器节点；且未显式设置CLUSTER_ID，可能导致重启后集群异常。
3. 模式切换冲突：使用apache/kafka:3.7.0时，因Kraft模式配置未生效，容器默认尝试使用ZooKeeper模式，因此要求zookeeper.connect配置。

解决方案 #

修正后的Kafka服务配置 #

将kafka服务的环境变量调整为apache/kafka镜像支持的格式，同时完善Kraft模式的关键配置：

kafka:
  image: 'apache/kafka:latest'
  container_name: kafka-broker
  environment:
    - KAFKA_ENABLE_KRAFT=yes
    - ALLOW_PLAINTEXT_LISTENER=yes
    - BROKER_ID=1001
    - NODE_ID=1001
    - PROCESS_ROLES=broker,controller
    - CONTROLLER_LISTENER_NAMES=CONTROLLER
    - INTER_BROKER_LISTENER_NAME=CLIENT
    - LISTENERS=CLIENT://:29092,EXTERNAL://:9092,CONTROLLER://:9093
    - LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
    - ADVERTISED_LISTENERS=CLIENT://kafka:29092,EXTERNAL://localhost:9092
    - CONTROLLER_QUORUM_VOTERS=1001@kafka:9093
    - CLUSTER_ID=your-unique-cluster-id-here # 可通过kafka-storage.sh random-uuid生成
  ports:
    - '9092:9092'

关键修改说明 #

• 移除所有KAFKA_CFG_前缀，直接使用配置项的全大写名称（如PROCESS_ROLES替代KAFKA_CFG_PROCESS_ROLES）。
• 将CONTROLLER_QUORUM_VOTERS中的127.0.0.1改为容器名kafka，确保容器内部能正确解析控制器地址。
• 添加CLUSTER_ID，可通过命令docker run --rm apache/kafka:latest kafka-storage.sh random-uuid生成唯一ID，避免重启后集群状态异常。

验证步骤 #

1. 替换docker-compose中的kafka服务配置。
2. 执行docker-compose down -v清理旧容器和数据卷。
3. 执行docker-compose up -d启动服务。
4. 查看Kafka容器日志：docker logs kafka-broker，确认无配置错误，服务正常启动。

内容的提问来源于stack exchange，提问作者nonpoliticaltag