Windows环境Docker部署Kafka无法消费数据排查求助

Let’s walk through this systematically—since both your Python consumer and Conduktor are struggling to interact with the linuxhint topic, the root cause is almost certainly tied to Kafka broker configuration or client connection setup, not just your consumer code alone.

1. Verify Kafka Broker Listener Configuration (Critical!) #

Bitnami Kafka’s default listener settings often cause headaches on Windows Docker, because the broker might advertise its internal container IP instead of localhost (the address your Windows clients need to connect to).

• First, check your container’s listener environment variables:

  docker exec <your-kafka-container-name> env | grep -E "KAFKA_LISTENERS|KAFKA_ADVERTISED_LISTENERS"
  

• You should see output like this (adjust if yours doesn’t match):

  KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://0.0.0.0:9093
  KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://kafka:9093
  

  The PLAINTEXT advertised listener must point to localhost:9092 so your Windows clients (Python/Conduktor) get a reachable address. If it’s set to an internal container IP (like 172.x.x.x), clients will try to connect to that unreachable address, leading to silent failures.

• If the settings are wrong, update your Docker run command or docker-compose.yml to set these variables, then restart the broker.

2. Confirm the Topic & Partition 2 Have Data #

Your consumer might be stuck simply because there are no messages in partition 2 to read.

• Inside the Kafka container, run these commands to check the topic’s status:

  # Describe the topic to verify partition 2 exists and is healthy
  docker exec <your-kafka-container-name> kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic linuxhint
  
  # Check if partition 2 has any messages (look for max offset > 0)
  docker exec <your-kafka-container-name> kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server localhost:9092 --topic linuxhint --time -1 --partitions 2
  

• If the max offset is 0, there are no messages in the partition—your consumer will wait indefinitely until new messages are produced. Try sending a test message first using kafka-console-producer.sh.

3. Fix Your Python Consumer Code #

Your current code has a few gaps that might cause silent waiting:

• You’re using assign() but not setting auto_offset_reset, which defaults to latest. If there are no new messages after the current offset, the consumer will hang.
• Add explicit offset reset and seek to the start of the partition to ensure you read existing messages:

from kafka import KafkaConsumer
from kafka import TopicPartition

print('Making connection.')
# Add auto_offset_reset and explicit api_version (match your Kafka version)
consumer = KafkaConsumer(
    bootstrap_servers='localhost:9092',
    auto_offset_reset='earliest',
    api_version=(2, 8, 0)  # Replace with your actual Kafka version (check via kafka-server-start.sh --version)
)
print('Assigning Topic.')
tp = TopicPartition('linuxhint', 2)
consumer.assign([tp])
# Seek to the beginning of the partition to read existing messages
consumer.seek_to_beginning([tp])
print('Getting message.')
for message in consumer:
    print(f"OFFSET: {message.offset}\t MSG: {message.value.decode('utf-8')}")

4. Validate Network Connectivity #

Double-check that your Windows machine can reach the Kafka broker’s port:

• Use telnet localhost 9092 or nc -zv localhost 9092 (install Netcat if needed) to confirm the port is open.
• Verify your Docker port mapping is correct with docker ps—look for 0.0.0.0:9092->9092/tcp under the PORTS column for your Kafka container.

5. Troubleshoot Conduktor #

If Conduktor can’t read topic metadata or message counts, it’s almost always a listener configuration issue (step 1). Fix the KAFKA_ADVERTISED_LISTENERS first, then:

• In Conduktor’s connection settings, ensure the Bootstrap Servers are set to localhost:9092.
• Confirm the Security Protocol is set to PLAINTEXT (unless you configured SSL, which you didn’t mention).
• Refresh the topic list after fixing the broker configuration.

内容的提问来源于stack exchange，提问作者Łukasz Kastelik