Windows Docker环境下NiFi无法向Kafka发送消息求助

Hey there, let's dig into why your NiFi can't publish messages to Kafka and fix this step by step. From your logs and configs, the core issue is that Kafka clients (both NiFi and the console producer inside the Kafka container) can't fetch cluster metadata properly, leading to that timeout error. Here's how to fix it:

1. Fix Kafka's ZooKeeper Connection #

Right now your KAFKA_ZOOKEEPER_CONNECT uses your host's static IP, but in Docker Compose's default network, containers can communicate directly using service names. Change this line in your docker-compose.yml:

KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

This lets Kafka find ZooKeeper using its service name instead of relying on host IP routing.

2. Adjust Kafka Listener Configs for Container Networking #

Your current listener setup has a small issue with how it advertises the internal address. Since NiFi is in the same Docker network as Kafka, it needs to use Kafka's internal service address. Update these Kafka environment variables:

KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://192.168.1.50:9094
KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

By setting INSIDE://kafka:9092, you're telling clients in the same Docker network (like NiFi) to connect to Kafka using its service name and internal port.

3. Update NiFi's Kafka Processor Configs #

Head into your PublishKafka and ConsumeKafka processors in NiFi, and set the Kafka Bootstrap Servers to kafka:9092. Forget the host IP here—since NiFi is in the same Docker network, it can resolve the kafka service name directly.

4. Test Kafka Container Internal Messaging #

When you run the console producer inside the Kafka container, use the internal port to send messages:

kafka-console-producer.sh --broker-list localhost:9092 --topic MainIngestionTopic

This connects to the local instance of Kafka inside the container, which should work once the listener configs are fixed.

5. Restart All Services #

After making these config changes, restart your Docker Compose stack to apply everything:

docker-compose down
docker-compose up -d

Quick Checks to Ensure Everything Works #

• Verify all containers are on the same Docker network (run docker network inspect <your-compose-network> to confirm)
• Double-check that the topic name in NiFi matches MainIngestionTopic exactly—Kafka topics are case-sensitive!
• Make sure ZooKeeper is fully up and running before Kafka starts (check the ZooKeeper logs for "Started AdminServer on address" messages)

内容的提问来源于stack exchange，提问作者afenkner