重启Kafka与ZooKeeper容器时出现Cluster ID不匹配错误

问题场景 #

使用以下Docker Compose配置部署Confluent ZooKeeper与Kafka，首次运行一切正常，但关闭容器后再次启动时出现Cluster ID不匹配错误：

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - zookeeper-data:/var/lib/zookeeper/data
      - zookeeper-logs:/var/lib/zookeeper/logs
      
  broker:
    image: confluentinc/cp-server:7.4.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://192.168.13.184:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      #KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://192.168.13.184:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
    volumes:
      - broker-data:/var/lib/kafka/data

volumes:
  zookeeper-data:
  zookeeper-logs:
  broker-data:  

错误日志：

ERROR Fatal error during KafkaServer startup. Prepare to shutdown
(kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID
vmR9HtjMQ1GSCA2Q_ePbjA doesn't match stored clusterId
Some(0H-v-ckbTz6vl3tMv9D-YQ) in meta.properties. The broker is trying
to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:368)
at kafka.Kafka$.main(Kafka.scala:114)
at kafka.Kafka.main(Kafka.scala) [2023-07-06 10:57:47,457] INFO shutting down (kafka.server.KafkaServer) [2023-07-06
10:57:47,515] INFO [ZooKeeperClient Kafka server] Closing.
(kafka.zookeeper.ZooKeeperClient) [2023-07-06 10:57:47,627] INFO
Session: 0x10005d424560001 closed (org.apache.zookeeper.ZooKeeper)
[2023-07-06 10:57:47,627] INFO EventThread shut down for session:
0x10005d424560001 (org.apache.zookeeper.ClientCnxn) [2023-07-06
10:57:47,631] INFO [ZooKeeperClient Kafka server] Closed.
(kafka.zookeeper.ZooKeeperClient) [2023-07-06 10:57:47,653] INFO App
info kafka.server for 1 unregistered
(org.apache.kafka.common.utils.AppInfoParser) [2023-07-06
10:57:47,656] INFO shut down completed (kafka.server.KafkaServer)
[2023-07-06 10:57:47,656] ERROR Exiting Kafka due to fatal exception
during startup. (kafka.Kafka$)
kafka.common.InconsistentClusterIdException: The Cluster ID
vmR9HtjMQ1GSCA2Q_ePbjA doesn't match stored clusterId
Some(0H-v-ckbTz6vl3tMv9D-YQ) in meta.properties. The broker is trying
to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:368)
at kafka.Kafka$.main(Kafka.scala:114)
at kafka.Kafka.main(Kafka.scala) [2023-07-06 10:57:47,658] INFO shutting down (kafka.server.KafkaServer)

解决步骤 #

该错误是由于Broker本地meta.properties文件中存储的Cluster ID与ZooKeeper中记录的Cluster ID不一致导致的，修复步骤如下：

• 停止当前容器

  docker-compose down
  

• 定位并修改Broker的meta.properties文件

  1. 查询Broker数据卷的本地挂载路径：

     docker volume inspect broker-data | grep -A 1 Mountpoint
     

  2. 进入该路径，找到meta.properties文件：

     # 替换为上一步查到的路径，示例：/var/lib/docker/volumes/<broker-data-volume>/_data
     cd /path/to/broker-data/_data
     

  3. 编辑meta.properties，将cluster.id的值修改为错误日志中ZooKeeper对应的Cluster ID（示例中为vmR9HtjMQ1GSCA2Q_ePbjA）

• 重新启动容器

  docker-compose up -d
  

预防措施 #

• 不要单独删除ZooKeeper的数据卷，因为ZooKeeper存储了整个Kafka集群的元数据信息
• 如果需要重新搭建集群，需同时删除ZooKeeper和Broker的数据卷：

  docker-compose down -v
  

  之后再重新启动服务

内容的提问来源于stack exchange，提问作者TheMah