WSL环境Docker部署Kafka，Kafkajs创建Topic报Broker -1未找到错误

在Windows 10的WSL环境中用Docker部署Zookeeper与Kafka，通过Node.js的Kafkajs库尝试创建Topic时，遭遇KafkaJSBrokerNotFound: Broker -1 not found in the cached metadata错误，谷歌搜索未找到有效解决方法。

代码片段 #

const {Kafka} = require("kafkajs")

run();
async function run(){
    try
    {
         const kafka = new Kafka({
              "clientId": "myapp",
              "brokers" :["localhost:9092"]
         })

        const admin = kafka.admin();
        console.log("Connecting.....")
        await admin.connect()
        console.log("Connected!")
        //A-M, N-Z
        await admin.createTopics({
            "topics": [{
                "topic" : "Users"
            }]
        })
        console.log("Created Successfully!")
        await admin.disconnect();
    }
    catch(ex)
    {
        console.error(`Something bad happened ${ex}`)
    }
    finally{
        process.exit(0);
    }
}

程序输出 #

Connecting.....
Connected!
Something bad happened KafkaJSBrokerNotFound: Broker -1 not found in the cached metadata

docker-compose.yml配置 #

services:
  zookeeper:
    image: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181

  kafka:
    restart: always
    image: ches/kafka
    hostname: localhost
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 109.253.189.218
      ZOOKEEPER_IP: zookeeper
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      ALLOW_PLAINTEXT_LISTENER: 1
    
    depends_on:
      - zookeeper

DEBUG级别日志 #

Connected!
{"level":"DEBUG","timestamp":"2023-03-30T11:00:02.355Z","logger":"kafkajs","message":"[Connection] Request Metadata(key: 3, version: 2)","broker":"localhost:9092","clientId":"myapp","correlationId":3,"expectResponse":true,"size":23}
{"level":"DEBUG","timestamp":"2023-03-30T11:00:02.357Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 2)","broker":"localhost:9092","clientId":"myapp","correlationId":3,"size":40,"data":{"brokers":[],"clusterId":"FsTkMSOPTlaLiIc5Qn65Fg","controllerId":-1,"topicMetadata":[]}}
{"level":"DEBUG","timestamp":"2023-03-30T11:00:02.357Z","logger":"kafkajs","message":"[Connection] Request Metadata(key: 3, version: 2)","broker":"localhost:9092","clientId":"myapp","correlationId":4,"expectResponse":true,"size":23}
{"level":"DEBUG","timestamp":"2023-03-30T11:00:02.359Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 2)","broker":"localhost:9092","clientId":"myapp","correlationId":4,"size":40,"data":{"brokers":[],"clusterId":"FsTkMSOPTlaLiIc5Qn65Fg","controllerId":-1,"topicMetadata":[]}}
Something bad happened KafkaJSBrokerNotFound: Broker -1 not found in the cached metadata

错误原因与解决方案 #

核心问题 #

从DEBUG日志可见，Kafka返回的元数据中brokers数组为空、controllerId为-1，说明Kafka集群未正确初始化，或客户端无法通过Kafka广播的地址连接到Broker。

具体来说，你的docker-compose.yml中Kafka的KAFKA_ADVERTISED_HOST_NAME配置为公网IP109.253.189.218，但WSL内的Node.js程序是通过localhost:9092访问Kafka。Kafka会将公网IP返回给客户端，客户端尝试连接该公网IP时失败，导致无法获取有效Broker元数据。

修复步骤 #

1. 修改Kafka广告地址配置：
   将KAFKA_ADVERTISED_HOST_NAME改为localhost，或WSL内部IP（如172.x.x.x段），确保客户端能访问该地址。同时推荐使用更现代的Kafka镜像（如confluentinc/cp-kafka），ches/kafka镜像较老旧，存在兼容性问题。

   修改后的docker-compose.yml示例：

   services:
     zookeeper:
       image: confluentinc/cp-zookeeper:7.4.0
       environment:
         ZOOKEEPER_CLIENT_PORT: 2181
         ZOOKEEPER_TICK_TIME: 2000
       ports:
         - "2181:2181"
   
     kafka:
       image: confluentinc/cp-kafka:7.4.0
       depends_on:
         - zookeeper
       ports:
         - "9092:9092"
       environment:
         KAFKA_BROKER_ID: 1
         KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
         KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://kafka:29092
         KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://kafka:29092
         KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
         KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
         KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
   

2. 重启Kafka容器：
   执行命令重建容器：

   docker-compose down
   docker-compose up -d
   

3. 验证Kafka状态：
   进入Kafka容器，用内置工具检查Broker状态：

   docker exec -it <kafka-container-id> kafka-topics --list --bootstrap-server localhost:9092
   

   能正常执行则说明Broker已正确初始化。

4. 重新运行Node.js程序：
   确保代码中brokers配置为["localhost:9092"]，再次执行程序即可正常创建Topic。

内容的提问来源于stack exchange，提问作者jen