AKS中Spring微服务无法连接Kafka问题求助

问题背景 #

Spring微服务部署到AKS集群后，其余服务运行正常，但依赖Kafka的服务反复出现无法建立连接的报错，日志显示服务始终尝试连接localhost:9092。已通过Helm单独部署Kafka，Pod环境变量中能获取到Kafka的服务地址，但问题依旧。

当前配置 #

application.properties #

spring.kafka.consumer.group-id=jihed
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.bootstrap-servers=kafka:9092
spring.kafka.properties.security.protocol=SASL_PLAINTEXT
spring.kafka.properties.sasl.mechanism=PLAIN
spring.kafka.properties.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user1" password="Ut8uZDJzPR";
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.json.trusted.packages=*

spring.kafka.producer.bootstrap-servers=kafka:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

报错日志 #

2024-05-13T16:11:17.713Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:17.713Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:18.917Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:18.917Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:20.021Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:20.021Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:21.024Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:21.025Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:22.228Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:22.228Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:23.432Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:23.432Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:24.335Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:24.335Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:25.339Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:25.339Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:26.342Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:26.342Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:27.246Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:27.246Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:28.149Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:28.149Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-05-13T16:11:29.052Z  INFO 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2024-05-13T16:11:29.053Z  WARN 1 --- [payload-service] [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

触发报错的代码 #

报错发生在调用createAppTopic方法创建Kafka Topic时：

public AppTopic createAppTopic(AppTopic topic) throws TopicException {
    try {
        // Persist the AppSchema object if it's not already persisted
        if (topic.getSchema() != null && topic.getSchema().getId() == null) {
            AppSchema persistedSchema = appSchemaRepo.save(topic.getSchema());
            topic.setSchema(persistedSchema);
        }

        // Create or modify Kafka topic
        kafkaAdmin.createOrModifyTopics(TopicBuilder.name(topic.getName()).partitions(topic.getPartitions()).replicas(topic.getReplicas()).config(TopicConfig.RETENTION_MS_CONFIG, topic.getRetentionPeriod()).build());
        // Save the AppTopic entity
        return appTopicRepo.save(topic);
    } catch (Exception e) {
        throw new TopicException("Cannot create topic for this reason", e);
    }
}

Pod环境变量 #

Pod已正确获取到Kafka服务地址：

KAFKA_PORT_9092_TCP=tcp://10.0.148.52:9092
KAFKA_PORT_9092_TCP_PROTO=tcp
KAFKA_SERVICE_HOST=10.0.148.52
KAFKA_SERVICE_PORT=9092
KAFKA_SERVICE_PORT_TCP_CLIENT=9092
KAFKA_PORT_9092_TCP_PORT=9092
KAFKA_PORT_9092_TCP_ADDR=10.0.148.52

排查建议 #

• 补充KafkaAdmin的配置：当前仅配置了consumer和producer的bootstrap-servers，但KafkaAdmin使用独立的配置项spring.kafka.admin.bootstrap-servers，未配置时会默认使用localhost:9092，这是AdminClient连接localhost的核心原因。需在application.properties中添加：

  spring.kafka.admin.bootstrap-servers=kafka:9092
  spring.kafka.admin.properties.security.protocol=SASL_PLAINTEXT
  spring.kafka.admin.properties.sasl.mechanism=PLAIN
  spring.kafka.admin.properties.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user1" password="Ut8uZDJzPR";
  

• 验证Kafka服务的DNS解析：在Pod内执行nslookup kafka，确认是否能解析到环境变量中的IP（10.0.148.52）。若解析失败，检查Kafka服务与当前服务是否在同一namespace，或是否存在DNS配置异常。
• 确认Kafka的SASL配置一致性：检查Helm部署的Kafka是否启用了SASL_PLAIN认证，且服务端配置的用户名、密码与服务端一致。可通过查看Kafka的Pod配置或Helm values.yaml确认。
• 测试Kafka端口可达性：在Pod内执行telnet kafka 9092或nc -zv kafka 9092，验证是否能连接到Kafka的9092端口。若连接失败，检查Kafka Service是否正确暴露端口，以及是否有NetworkPolicy限制流量。
• 检查配置加载情况：在Pod内查看实际生效的配置文件，确认application.properties中的bootstrap-servers是否被正确读取，是否存在环境变量（如SPRING_KAFKA_ADMIN_BOOTSTRAP_SERVERS）覆盖配置的情况。

内容的提问来源于stack exchange，提问作者Jihed Ben Zarb