如何解决Kafka中无法连接localhost:9092节点的问题？

问题场景 #

使用以下docker-compose.yml部署包含Zookeeper、Kafka、MySQL的集群：

version: '2.1'

networks:
  app-tier:
    driver: bridge

services:

  zookeeper:
    image: 'bitnami/zookeeper:3'
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
      - app-tier

  kafka:
    image: 'bitnami/kafka:2'
    container_name: kafkaContainer
    ports:
      - '9092:9092'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_CFG_LISTENERS=PLAINTEXT://kafkaContainer:29092,PLAINTEXT_HOST://kafkaContainer:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafkaContainer:29092,PLAINTEXT_HOST://localhost:9092
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
    depends_on:
      - "zookeeper"
    links:
      - zookeeper
    healthcheck:
      test:
        ["CMD", "kafka-topics.sh", "--list", "--zookeeper", "zookeeper:2181"]
      interval: 30s
      timeout: 10s
      retries: 4
    networks:
      - app-tier

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: 'db'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'user123'
      MYSQL_ROOT_PASSWORD: 'user123'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    volumes:
      - my-db:/var/lib/mysql
    depends_on:
      - "kafka"
    links:
      - kafka

volumes:
  my-db:

操作现象 #

• Topic创建与查看正常：
  执行命令：

  docker exec -it kafkaContainer kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3  --topic test1_skh
  

  返回：

  WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. Created topic test1_skh.

  执行列表命令：

  docker exec -it kafkaContainer kafka-topics.sh --list --zookeeper zookeeper:2181
  

  返回：

  test1_skh

• 生产者连接localhost:9092报错：
  执行命令：

  docker exec -it kafkaContainer kafka-console-producer.sh --topic test1_skh --bootstrap-server localhost:9092
  

  返回错误：

  [2022-07-28 22:25:34,609] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

• 改用kafka:9092作为bootstrap-server，可发送第一条消息但后续仍报相同错误。

• 消费者连接localhost:9092也报错：
  执行命令：

  docker exec -it kafkaContainer kafka-console-consumer.sh --topic test1_skh --from-beginning --bootstrap-server localhost:9092
  

  返回错误：

  [2022-07-28 22:28:26,844] WARN [Consumer clientId=consumer-console-consumer-91144-1, groupId=console-consumer-91144] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

解决方案 #

1. 修正Kafka监听配置 #

问题核心是Kafka的listeners配置限制了仅能通过容器名kafkaContainer访问，修改kafka服务的environment中的KAFKA_CFG_LISTENERS为监听所有接口：

KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092

此配置让Kafka监听容器内所有可用IP，避免因绑定固定容器名导致的访问限制。

2. 使用正确的bootstrap-server地址 #

容器内部的生产者/消费者需要使用Kafka容器内部的监听地址，即kafkaContainer:29092（对应配置中的PLAINTEXT监听器），而非localhost:9092或kafka:9092（容器名是kafkaContainer，不是kafka）。

3. 重启服务 #

执行命令重启整个集群：

docker-compose down && docker-compose up -d

验证操作 #

• 启动生产者：

  docker exec -it kafkaContainer kafka-console-producer.sh --topic test1_skh --bootstrap-server kafkaContainer:29092
  

  输入消息后可正常发送，无连接错误。

• 启动消费者：

  docker exec -it kafkaContainer kafka-console-consumer.sh --topic test1_skh --from-beginning --bootstrap-server kafkaContainer:29092
  

  可正常接收之前发送的消息。

内容的提问来源于stack exchange，提问作者luismir15