如何将Kubernetes中Kafka Kraft模式的副本因子改为3？

一、更新StatefulSet配置 #

需要调整副本数、节点ID配置、控制器集群投票列表及副本因子参数，修改后的完整配置如下：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-statefulset
  namespace: kafka
  labels:
    app: kafka-cluster
spec:
  serviceName: kafka-svc
  replicas: 3  # 调整为3副本
  selector:
    matchLabels:
      app: kafka-cluster
  template:
    metadata:
      labels:
        app: kafka-cluster
    spec:
      containers:
        - name: kafka-container
          image: 'bitnami/kafka:latest'
          ports:
            - containerPort: 9092
            - containerPort: 9093
          env:
            # 通过Pod hostname自动获取节点ID，适配StatefulSet有序命名
            - name: KAFKA_BROKER_ID_COMMAND
              value: "echo $(hostname | cut -d'-' -f3)"
            - name: KAFKA_CFG_NODE_ID_COMMAND
              value: "echo $(hostname | cut -d'-' -f3)"
            - name: KAFKA_ENABLE_KRAFT
              value: "yes"
            - name: KAFKA_CFG_PROCESS_ROLES
              value: "broker,controller"
            - name: KAFKA_CFG_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CFG_LISTENERS
              value: "CLIENT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094"
            - name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP
              value: "CONTROLLER:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT"
            - name: KAFKA_CFG_INTER_BROKER_LISTENER_NAME
              value: "CLIENT"
            # 动态生成广告监听地址，适配每个Pod的hostname
            - name: KAFKA_CFG_ADVERTISED_LISTENERS
              value: "CLIENT://${HOSTNAME}.kafka-svc.kafka.svc.cluster.local:9092,EXTERNAL://127.0.0.1:9094"
            # 配置3节点控制器集群投票列表
            - name: KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-statefulset-0.kafka-svc.kafka.svc.cluster.local:9093,1@kafka-statefulset-1.kafka-svc.kafka.svc.cluster.local:9093,2@kafka-statefulset-2.kafka-svc.kafka.svc.cluster.local:9093"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
            # 修改默认副本因子为3
            - name: KAFKA_DEFAULT_REPLICATION_FACTOR
              value: "3"
            # 修改offsets主题副本因子为3
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            # 可选：修改事务日志主题副本因子为3
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "3"
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"

关键修改点说明 #

• spec.replicas: 从1改为3，启动3个Kafka节点
• 移除固定KAFKA_BROKER_ID和KAFKA_CFG_NODE_ID，改用*_COMMAND通过Pod hostname自动获取节点序号（对应0、1、2）
• KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 配置3个控制器节点的地址，格式为节点ID@Pod域名:控制器端口
• KAFKA_CFG_ADVERTISED_LISTENERS: 使用${HOSTNAME}动态生成每个Pod的内部访问地址
• 调整副本因子相关参数为3，确保新建主题默认使用3副本，内部系统主题也保持高可用

二、更新Headless Service配置 #

新增控制器端口（9093）的暴露，确保控制器节点之间能正常通信：

apiVersion: v1
kind: Service
metadata:
  name: kafka-svc
  labels:
    app: kafka-cluster
spec:
  clusterIP: None
  ports:
    - name: client
      port: 9092
      protocol: TCP
      targetPort: 9092
    - name: controller
      port: 9093
      protocol: TCP
      targetPort: 9093
  selector:
    app: kafka-cluster

三、部署操作说明 #

1. 备份原有Kafka数据（若需保留）
2. 删除原有StatefulSet：kubectl delete statefulset kafka-statefulset -n kafka
3. 应用更新后的配置：

   kubectl apply -f updated-statefulset.yaml
   kubectl apply -f updated-headless-svc.yaml
   

4. 等待3个Pod全部启动并进入Running状态，验证集群状态：

   kubectl exec -it kafka-statefulset-0 -n kafka -- kafka-topics.sh --list --bootstrap-server localhost:9092
   kubectl exec -it kafka-statefulset-0 -n kafka -- kafka-cluster.sh --describe --bootstrap-server localhost:9092
   

内容的提问来源于stack exchange，提问作者Peter S.