如何在Kubernetes上部署多副本Kafka并解决扩容异常问题

问题根因 #

• Broker ID重复：Kafka 要求集群内每个节点必须拥有全局唯一的broker.id，普通无状态Deployment未给副本分配固定唯一标识，扩容、Pod重启时如果使用固定ID配置、或者ID随机生成逻辑未持久化，就会触发ID冲突报错。
• advertised.listeners 地址无法解析：该配置需要填写客户端可稳定访问的地址，普通Deployment的Pod IP是动态生成的，若未绑定固定DNS规则，配置的地址会随着Pod重建失效，导致解析失败。

注：Kafka 属于有状态服务，使用StatefulSet配合Headless Service部署是更稳定的方案，以下提供可直接复用的配置示例，如果你必须使用Deployment，可参考核心配置逻辑调整。

# 第一步：创建Headless Service，用于Pod稳定DNS解析
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  labels:
    app: kafka
spec:
  ports:
  - port: 9092
    name: internal
  clusterIP: None
  selector:
    app: kafka
---
# 第二步：创建ZooKeeper服务（Kafka依赖，若已有可跳过）
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: zookeeper
        image: bitnami/zookeeper:3.8
        ports:
        - containerPort: 2181
        env:
        - name: ALLOW_ANONYMOUS_LOGIN
          value: "yes"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  ports:
  - port: 2181
  selector:
    app: zookeeper
---
# 第三步：Kafka StatefulSet配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-headless
  replicas: 3 # 可按需调整副本数，不会出现ID重复
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: bitnami/kafka:2.8.1
        ports:
        - containerPort: 9092
          name: internal
        env:
        - name: KAFKA_BROKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['apps.kubernetes.io/pod-index'] # 直接取StatefulSet副本序号作为Broker ID，全局唯一
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zookeeper:2181"
        - name: KAFKA_LISTENERS
          value: "INTERNAL://0.0.0.0:9092"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "INTERNAL://$(POD_NAME).kafka-headless.default.svc.cluster.local:9092"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "INTERNAL:PLAINTEXT"
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: "INTERNAL"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: kafka-data
          mountPath: /bitnami/kafka
  volumeClaimTemplates: # 持久化存储，避免数据丢失
  - metadata:
      name: kafka-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

• 唯一Broker ID生成：直接使用StatefulSet自带的副本序号作为KAFKA_BROKER_ID，扩容时序号自动递增，不会出现重复问题
• 稳定可解析的advertised.listeners：通过$(POD_NAME).kafka-headless.default.svc.cluster.local格式配置，K8s会自动解析到对应Pod的IP，Pod重启后DNS映射自动更新，不会出现解析失败
• 如果需要暴露给集群外访问，可额外配置NodePort/LoadBalancer类型的Service，在KAFKA_ADVERTISED_LISTENERS中添加外部访问地址即可

内容的提问来源于stack exchange，提问作者Jack