使用KafkaJS库的Node.js Kafka生产者无法发送消息排查

Problem Description #

I've written a Node.js Kafka producer using KafkaJS, here's the sample code (KafkaJS-Producer.js):

const { Kafka } = require('kafkajs')
const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
})
const producer = kafka.producer()

async () => {
  await producer.connect()
  await producer.send({
    topic: 'topic-name',
    messages: [
      { key: 'key1', value: 'hello world' },
      { key: 'key2', value: 'hey hey!' }
    ],
  })
  await producer.disconnect()
}

When I run node SampleProducer.js, there's no error, but when I try to consume the topic with:

kafka-console-consumer --bootstrap-server localhost:9092 --topic topic-name --from-beginning

I get no messages. Kafka is running locally and the topic is already created. What could be the issue?

Troubleshooting & Fixes #

Let's break down the most likely causes, starting with the obvious one in your code:

• You defined an async function but never executed it
  This is the biggest red flag here. You created an anonymous async function but didn't call it—so none of the producer logic (connect, send, disconnect) actually runs. That's why you see no errors (the function never runs to throw any) but also no messages.
  Fix this by either turning it into an immediately-invoked function expression (IIFE):

  (async () => {
    await producer.connect()
    await producer.send({
      topic: 'topic-name',
      messages: [
        { key: 'key1', value: 'hello world' },
        { key: 'key2', value: 'hey hey!' }
      ],
    })
    await producer.disconnect()
  })()
  

  Or assign it to a variable and call it explicitly:

  const runProducer = async () => {
    await producer.connect()
    await producer.send({ /* ... */ })
    await producer.disconnect()
  }
  
  runProducer()
  

• Missing error handling hides silent failures
  Even if you fix the function execution, without error handling you might miss issues like failed Kafka connections, topic authorization problems, or message send failures. Wrap your logic in a try/catch block to surface any hidden errors:

  (async () => {
    try {
      await producer.connect()
      console.log('Producer connected successfully')
      
      const sendResult = await producer.send({
        topic: 'topic-name',
        messages: [
          { key: 'key1', value: 'hello world' },
          { key: 'key2', value: 'hey hey!' }
        ],
      })
      console.log('Messages sent successfully:', sendResult)
      
      await producer.disconnect()
    } catch (error) {
      console.error('Producer encountered an error:', error)
      // Clean up connection even on failure
      await producer.disconnect().catch(err => console.error('Disconnect failed:', err))
    }
  })()
  

  This will log exactly what's going wrong if the producer can't send messages.

• Verify producer configuration details
  Double-check that:

  • Your brokers array matches the actual Kafka listener address (if Kafka is configured to listen on a different port/host than localhost:9092, this will fail silently without error handling).
  • The topic name topic-name is exactly the same as the one you created (topic names are case-sensitive in most Kafka setups).

• Check Kafka topic replication and ack settings
  By default, KafkaJS uses acks: 1 (wait for the leader broker to acknowledge the write). If your topic has multiple replicas but followers are offline, the send might hang or fail silently. You can temporarily set acks: 0 (no acknowledgment required) to test if messages get through (not recommended for production), or ensure all replicas are online if you're using a multi-node cluster.

内容的提问来源于stack exchange，提问作者Aftab