部署在Kubernetes上的Spring Boot应用无法连接Kafka问题

云原生阿岚

2026-6-13

问题描述

开发了集成Kafka的Spring Boot应用，部署到Kubernetes集群后，应用无法解析Kafka集群的bootstrap URLs，启动失败。

错误日志

org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:326) ~[spring-context-6.2.3.jar!/:6.2.3]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:510) ~[spring-context-6.2.3.jar!/:6.2.3]
    at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:295) ~[spring-context-6.2.3.jar!/:6.2.3]
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:240) ~[spring-context-6.2.3.jar!/:6.2.3]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:1006) ~[spring-context-6.2.3.jar!/:6.2.3]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:630) ~[spring-context-6.2.3.jar!/:6.2.3]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.4.3.jar!/:3.4.3]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752) ~[spring-boot-3.4.3.jar!/:3.4.3]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) ~[spring-boot-3.4.3.jar!/:3.4.3]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:318) ~[spring-boot-3.4.3.jar!/:3.4.3]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1361) ~[spring-boot-3.4.3.jar!/:3.4.3]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1350) ~[spring-boot-3.4.3.jar!/:3.4.3]
    at fr.formationacademy.scpiinvestpluspartner.ScpiInvestPlusPartnerApplication.main(ScpiInvestPlusPartnerApplication.java:10) ~[!/:0.0.1-SNAPSHOT]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[na:na]
    at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102) ~[scpi-invest-plus-partner.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64) ~[scpi-invest-plus-partner.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40) ~[scpi-invest-plus-partner.jar:0.0.1-SNAPSHOT]

Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.<init>(LegacyKafkaConsumer.java:265) ~[kafka-clients-3.8.1.jar!/:na]
    at org.apache.kafka.clients.consumer.internals.ConsumerDelegateCreator.create(ConsumerDelegateCreator.java:65) ~[kafka-clients-3.8.1.jar!/:na]
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:600) ~[kafka-clients-3.8.1.jar!/:na]
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:595) ~[kafka-clients-3.8.1.jar!/:na]
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory$ExtendedKafkaConsumer.<init>(DefaultKafkaConsumerFactory.java:498) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createRawConsumer(DefaultKafkaConsumerFactory.java:453) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:430) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumerWithAdjustedProperties(DefaultKafkaConsumerFactory.java:407) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:374) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumer(DefaultKafkaConsumerFactory.java:335) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.<init>(KafkaMessageListenerContainer.java:876) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer.doStart(KafkaMessageListenerContainer.java:387) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:520) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.listener.ConcurrentMessageListenerContainer.doStart(ConcurrentMessageListenerContainer.java:264) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:520) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.config.KafkaListenerEndpointRegistry.startIfNecessary(KafkaListenerEndpointRegistry.java:436) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.kafka.config.KafkaListenerEndpointRegistry.start(KafkaListenerEndpointRegistry.java:382) ~[spring-kafka-3.3.3.jar!/:3.3.3]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:323) ~[spring-context-6.2.3.jar!/:6.2.3]
    ... 20 common frames omitted

Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:103) ~[kafka-clients-3.8.1.jar!/:na]
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:62) ~[kafka-clients-3.8.1.jar!/:na]
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:58) ~[kafka-clients-3.8.1.jar!/:na]
    at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.<init>(LegacyKafkaConsumer.java:184) ~[kafka-clients-3.8.1.jar!/:na]
    ... 37 common frames omitted

Kubernetes Kafka配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: confluentinc/cp-kafka:6.1.1
          ports:
            - containerPort: 9092
            - containerPort: 9094
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

            - name: KAFKA_LISTENERS
              value: "INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: "INTERNAL://$(POD_NAME).kafka-headless:9092,OUTSIDE://$(POD_NAME).kafka-svc:9094"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT"
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "3"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "INTERNAL"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper:2181"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
    - name: internal-port
      protocol: TCP
      port: 9092
      targetPort: 9092
    - name: outside-port
      protocol: TCP
      port: 9094
      targetPort: 9094

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-svc
spec:
  selector:
    app: kafka
  ports:
    - name: internal-port
      protocol: TCP
      port: 9092
      targetPort: 9092
    - name: outside-port
      protocol: TCP
      port: 9094
      targetPort: 9094
  type: ClusterIP

ZooKeeper配置

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zookeeper
          image: wurstmeister/zookeeper
          ports:
            - containerPort: 2181
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  selector:
    app: zookeeper
  ports:
    - protocol: TCP
      port: 2181
      targetPort: 2181

Spring Boot application.yml配置

spring:
  application:
    version: 1.0.0
    name: scpi-invest-plus-partner
  kafka:
    bootstrap-servers: kafka-0.kafka-headless:9092,kafka-1.kafka-headless:9092,kafka-2.kafka-headless:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: scpi-partner-group
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      properties:
        spring.json.trusted.packages: "*"

解决方案

1. 替换为Headless Service作为Bootstrap地址

当前写死了每个Kafka Pod的地址，这种方式可靠性差——StatefulSet是逐个启动Pod的，Spring应用启动时可能部分Kafka Pod还未就绪，导致DNS解析失败。

直接使用Kafka的headless service域名，Kafka客户端会自动发现集群内所有可用节点：

spring:
  kafka:
    bootstrap-servers: kafka-headless:9092

2. 等待Kafka集群就绪后再启动Spring应用

在Spring应用的Deployment中添加initContainer，确保Kafka集群完全可用后再启动应用：

initContainers:
  - name: wait-for-kafka
    image: confluentinc/cp-kafka:6.1.1
    command: ["sh", "-c", "until kafka-topics --list --bootstrap-server kafka-headless:9092; do echo waiting for kafka; sleep 2; done"]

3. 检查Namespace一致性

如果Spring应用和Kafka不在同一个Kubernetes Namespace，需要在bootstrap地址中补充Namespace信息：

spring:
  kafka:
    bootstrap-servers: kafka-headless.<kafka-namespace>.svc.cluster.local:9092

比如Kafka部署在kafka namespace，地址就写kafka-headless.kafka.svc.cluster.local:9092。

4. 验证DNS解析和网络连通性

进入Spring应用的Pod，执行以下命令排查问题：

解析headless service：nslookup kafka-headless，确认能返回Kafka Pod的IP列表
测试端口连通性：telnet kafka-0.kafka-headless 9092，确认能连接到Kafka节点

内容的提问来源于stack exchange，提问作者Speedkillsx

部署在Kubernetes上的Spring Boot应用无法连接Kafka问题

错误日志 #

Kubernetes Kafka配置 #

ZooKeeper配置 #

Spring Boot application.yml配置 #

1. 替换为Headless Service作为Bootstrap地址 #

2. 等待Kafka集群就绪后再启动Spring应用 #

3. 检查Namespace一致性 #

4. 验证DNS解析和网络连通性 #