远程Linux服务器K8s部署Kafka遇Pod崩溃及配置疑问

问题背景 #

远程Kubernetes集群中ZooKeeper Pod已部署成功，但Kafka Pod始终处于CrashLoopBackoff状态，同时对Deployment中KAFKA_ADVERTISED_HOST_NAME的取值存在困惑。

使用的Kubernetes资源文件 #

Kafka Service #

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "0"
  type: LoadBalancer

Kafka Deployment #

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kafka-broker0
spec:
  replicas: 2
  selector:
    matchLabels:
        app: kafka
        id: "0"
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.1.240
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_CREATE_TOPICS
          value: admintome-test:1:1

kubectl describe pod 输出 #

Name:             kafka-broker0-7cf7d4559-8frpn
Namespace:        default
Priority:         0
Service Account:  default
Node:             prd-stg-185/172.0.0.1
Start Time:       Fri, 18 Nov 2022 13:05:12 +0400
Labels:           app=kafka
                  id=0
                  pod-template-hash=7cf7d4559
Annotations:      cni.projectcalico.org/containerID: 82fe4ebcf2b634bd29b3a2c95292f70493c3ad7cf3dc80369fc9e95be619da34
                  cni.projectcalico.org/podIP: 192.168.123.0/32
                  cni.projectcalico.org/podIPs: 192.168.123.0/32
Status:           Running
IP:               192.168.123.0
IPs:
  IP:           192.168.123.0
Controlled By:  ReplicaSet/kafka-broker0-7cf7d4559
Containers:
  kafka:
    Container ID:   containerd://45c9ae210cbb6a5b29b87ea3bc3041f10c9132908fa4eba27a7a7f480b40c08e
    Image:          wurstmeister/kafka
    Image ID:       docker.io/wurstmeister/kafka@sha256:2d4bbf9cc83d9854d36582987da5f939fb9255fb128d18e3cf2c6ad825a32751
    Port:           9092/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 21 Nov 2022 09:36:45 +0400
      Finished:     Mon, 21 Nov 2022 09:37:08 +0400
    Ready:          False
    Restart Count:  752
    Environment:
      KAFKA_ADVERTISED_PORT:       30718
      KAFKA_ADVERTISED_HOST_NAME:  192.168.1.240
      KAFKA_ZOOKEEPER_CONNECT:     zoo1:2181
      KAFKA_BROKER_ID:             0
      KAFKA_CREATE_TOPICS:         admintome-test:1:1
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-66qch (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-66qch:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                        From     Message
  ----     ------   ----                       ----     -------
  Warning  BackOff  6m16s (x17583 over 2d20h)  kubelet  Back-off restarting failed container

问题排查与修复建议 #

1. 先获取容器日志定位核心错误 #

当前describe pod仅显示重启事件，无具体退出原因，执行以下命令拿日志：

# 实时查看当前容器日志
kubectl logs kafka-broker0-7cf7d4559-8frpn -f
# 如果容器已终止，查看上一次启动日志
kubectl logs kafka-broker0-7cf7d4559-8frpn --previous

2. 修正KAFKA_ADVERTISED_HOST_NAME及相关配置 #

你的配置存在几个关键问题：

• Broker ID重复：Deployment设置了2个副本，但所有Pod的KAFKA_BROKER_ID都为0，Kafka集群要求Broker ID唯一，这会直接导致启动失败。建议使用StatefulSet而非Deployment部署多Broker Kafka，StatefulSet可自动分配唯一的序号作为Broker ID。
• KAFKA_ADVERTISED_HOST_NAME 取值不当：
  • 集群内部访问：应使用Kafka Service的名称kafka-service，Pod重启或IP变化时不影响通信。
  • 外部访问：使用LoadBalancer的外部IP，而非固定节点IP（除非采用NodePort模式并绑定固定节点）。
  • 推荐使用KAFKA_ADVERTISED_LISTENERS替代单独的KAFKA_ADVERTISED_HOST_NAME和KAFKA_ADVERTISED_PORT，配置更清晰：

    - name: KAFKA_ADVERTISED_LISTENERS
      value: "PLAINTEXT://kafka-service:9092,PLAINTEXT_HOST://192.168.1.240:30718"
    - name: KAFKA_LISTENERS
      value: "PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:30718"
    

• 端口映射不匹配：Service的port为9092，但Deployment中KAFKA_ADVERTISED_PORT设为30718，需确认Service是否配置了NodePort或LoadBalancer端口映射到30718，否则外部无法访问。

3. 其他潜在问题排查 #

• ZooKeeper连通性：验证Kafka Pod能否访问zoo1:2181，可在容器内执行nc -zv zoo1 2181测试（若容器未安装nc，可临时进入容器安装：kubectl exec -it kafka-broker0-7cf7d4559-8frpn -- apt update && apt install netcat -y）。
• 资源限制：当前Pod未设置CPU和内存限制，可能因资源不足导致重启，建议添加资源配置：

  resources:
    requests:
      memory: "512Mi"
      cpu: "500m"
    limits:
      memory: "1Gi"
      cpu: "1"
  

内容的提问来源于stack exchange，提问作者Gøvi