Spring Boot微服务Kafka AdminClientConfig忽略配置连接异常排查

我有一个基于Spring Boot的微服务应用，为实现服务间事件驱动通信集成了Kafka。所有服务与cp-kafka均部署在docker-compose的同一桥接网络中。启动docker-compose后，生产者和消费者均无法连接到Broker：AdminClientConfig始终使用localhost:9092，而非Broker配置中指定的kafka:9092。

生产者日志 #

2023-02-14 13:09:17.563  INFO [article-service,,] 1 --- [           main] o.a.k.clients.admin.AdminClientConfig    : AdminClientConfig values: 
2023-02-14T13:09:17.563482000Z  bootstrap.servers = [localhost:9092]
2023-02-14T13:09:17.563518700Z  client.dns.lookup = use_all_dns_ips
2023-02-14T13:09:17.563524100Z  client.id = 
2023-02-14T13:09:17.563528200Z  connections.max.idle.ms = 300000
...

消费者日志 #

初始连接配置正确：

2023-02-14 13:10:13.358  INFO 1 --- [           main] o.a.k.clients.consumer.ConsumerConfig    : ConsumerConfig values: 
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [kafka:9092]
        check.crcs = true
        client.dns.lookup = use_all_dns_ips
        client.id = consumer-saveArticle-1
        client.rack = 
        connections.max.idle.ms = 540000
...

重试时切换为AdminClientConfig并使用localhost:9092：

2023-02-14 13:10:42.365  INFO 1 --- [   scheduling-1] o.a.k.clients.admin.AdminClientConfig    : AdminClientConfig values: 
        bootstrap.servers = [localhost:9092]
        client.dns.lookup = use_all_dns_ips
        client.id = 
        connections.max.idle.ms = 300000

相关配置 #

docker-compose.yml片段 #

...
networks:
  backend:
    name: backend
    driver: bridge

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    container_name: dev.infra.zookeeper
    networks:
      - backend
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.3.0
    container_name: kafka
    networks:
      - backend
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      AUTO_CREATE_TOPICS_ENABLE: 'false'
...

生产者application.yml #

spring:
  kafka:
    producer:
      bootstrap-servers: kafka:9092
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
    topic:
      name: saveArticle

消费者application.yml #

spring:
  kafka:
    consumer:
      bootstrap-servers: kafka:9092
      auto-offset-reset: earliest
      group-id: stock
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    topic:
      name: saveArticle

依赖配置 #

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>3.0.2</version>
    <type>pom</type>
</dependency>

请问localhost:9092的配置来源是什么？为何会忽略Broker配置中指定的kafka:9092？如何解决该问题？

1. localhost:9092的配置来源 #

localhost:9092是Kafka客户端的默认配置值。当Spring Kafka无法从自定义配置中获取到有效的bootstrap-servers时，就会回退使用这个默认地址。

2. 为何忽略指定的kafka:9092 #

你仅在producer和consumer节点下配置了bootstrap-servers，但AdminClient（用于主题管理、元数据查询等后台操作）不会继承生产者/消费者的配置。Spring Kafka中，AdminClient的配置需要单独在全局节点或专属节点中设置，否则会使用默认值。

3. 解决方法 #

方法一：添加全局Kafka配置 #

在生产者和消费者的application.yml中，添加全局的bootstrap-servers，让所有Kafka客户端（生产者、消费者、AdminClient）共享该配置：

spring:
  kafka:
    bootstrap-servers: kafka:9092 # 全局配置，覆盖所有客户端的默认值
    producer:
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
    topic:
      name: saveArticle

消费者端同理添加全局配置，可保留或删除consumer节点下的重复配置，全局配置优先级更高。

方法二：单独配置AdminClient #

如果需要给AdminClient设置独立配置，可在application.yml中添加admin节点：

spring:
  kafka:
    admin:
      bootstrap-servers: kafka:9092 # 单独为AdminClient指定地址
    producer:
      bootstrap-servers: kafka:9092
      ...
    consumer:
      bootstrap-servers: kafka:9092
      ...

额外检查点 #

• 确认docker-compose中所有服务都加入了backend网络，且Kafka容器名称为kafka，确保容器间能通过名称解析地址。
• 检查是否存在环境变量KAFKA_BOOTSTRAP_SERVERS被设置为localhost:9092，该环境变量会覆盖配置文件中的值。

内容的提问来源于stack exchange，提问作者Specialized