如何解决K8s集群中Spring Boot应用连接Kafka Pod的网络问题？

我有一个K8s集群，包含运行Spring Boot应用的core-pod和Kafka Pod，两者都配置了Service，但启动应用Pod时出现连接问题。之前在普通Docker环境中能成功连接这两个容器。

应用配置yml #

server:
  port: 8080
logging:
  level:
    nuclear.bot.core: TRACE
notification:
  level: INFO
spring:
  kafka:
    template:
      default-topic: PARSER.EVENT
    consumer:
      bootstrap-servers: kafka-service:9092
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: nuclear.bot.core.config.MessageAdapter
      group-id: core
    properties:
      key-class: java.lang.String
      value-class: nuclear.bot.core.dto.AgentMessage

应用尝试连接kafka-service:9092。

应用Pod配置yml #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: core
  labels:
    app: core
spec:
  replicas: 1
  selector:
    matchLabels:
      app: core-pod
  template:
    metadata:
      name: core-pod
      labels:
        app: core-pod
    spec:
      containers:
        - name: nuclear-bot-core
          image: vladi15151/nuclear-bot-core:0.9.7
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP

Kafka的Pod与Service配置yml #

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  selector:
    app: kafka-broker
  ports:
    - name: kafka-port
      port: 9092
      targetPort: 9092
      nodePort: 30126
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kafka-broker
  name: kafka-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      hostname: kafka-broker
      containers:
        - image: bitnami/kafka
          imagePullPolicy: IfNotPresent
          name: kafka-broker
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper-service:2181"
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://localhost:9092
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://localhost:9092
            # Creates a topic with one partition and one replica.
            - name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
              value: "true"
            - name: KAFKA_CREATE_TOPICS
              value: "PARSEREVENT:1:1"
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"

错误日志 #

2023-06-09 21:53:26.193  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:27.249  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:27.249  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:27.249  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:28.205  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:28.205  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:28.205  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:29.309  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:29.309  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:29.309  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:30.113  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:30.113  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:30.113  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected
2023-06-09 21:53:31.143  INFO 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Node -1 disconnected.
2023-06-09 21:53:31.143  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Connection to node -1 (kafka-service/10.99.46.194:9092) could not be established. Broker may not be available.
2023-06-09 21:53:31.143  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-core-1, groupId=core] Bootstrap broker kafka-service:9092 (id: -1 rack: null) disconnected

1. 修正Kafka监听配置 #

当前Kafka仅监听localhost:9092，只能接受容器内部连接，集群内其他Pod无法通过Service访问。修改Kafka Deployment的环境变量：

• 将KAFKA_LISTENERS改为PLAINTEXT://0.0.0.0:9092，让Kafka监听容器所有网络接口
• 将KAFKA_ADVERTISED_LISTENERS改为PLAINTEXT://kafka-service:9092，告知客户端通过Service地址连接Kafka

修改后的环境变量片段：

env:
  - name: KAFKA_BROKER_ID
    value: "1"
  - name: KAFKA_ZOOKEEPER_CONNECT
    value: "zookeeper-service:2181"
  - name: KAFKA_LISTENERS
    value: PLAINTEXT://0.0.0.0:9092
  - name: KAFKA_ADVERTISED_LISTENERS
    value: PLAINTEXT://kafka-service:9092
  # 其余环境变量保持不变

2. 验证网络连通性 #

• 进入core-pod内部，执行telnet kafka-service 9092或nc -zv kafka-service 9092，测试端口连通性
• 执行kubectl describe service kafka-service，确认Endpoints是否正确关联到Kafka Pod
• 用kubectl get pods检查Kafka Pod状态，kubectl logs <kafka-pod-name>查看Kafka启动日志，确认端口绑定正常

3. 检查Spring Boot配置与网络策略 #

• 确认spring.kafka.consumer.bootstrap-servers配置为kafka-service:9092，与Service名称、端口一致
• 若集群启用了NetworkPolicy，检查是否存在阻止core-pod与Kafka Service通信的规则，必要时添加允许通信的策略

4. 确认ZooKeeper状态 #

Kafka依赖ZooKeeper，执行kubectl get pods查看ZooKeeper Pod状态，kubectl logs <zookeeper-pod-name>确认ZooKeeper正常启动。

内容的提问来源于stack exchange，提问作者Roberto