跨机器部署的Kafka多Broker集群无法正常工作

Let’s break down the key checks and fixes to get your cross-machine Kafka cluster up and running, starting with the gaps in your provided configuration and common pitfalls.

1. First, Fix Machine 2's Kafka server.properties #

Your machine 2 Kafka config is incomplete, and this is almost certainly a critical issue. Here’s what you need to set to match the cluster requirements:

• Unique broker.id: Must be different from machine 1’s 1 — set this to 2
• Port: Use a unique port (e.g., 9095) or ensure port 9094 isn’t in use on machine 2
• ZooKeeper connection string: Exact same as machine 1: 10.0.215.22:2181,10.0.229.48:2181
• Match critical flags: Keep delete.topic.enable=true and auto.create.topics.enable=true consistent with machine 1

Sample critical config snippet for machine 2:

broker.id=2
port=9095
zookeeper.connect=10.0.215.22:2181,10.0.229.48:2181
delete.topic.enable=true
auto.create.topics.enable=true
num.partitions=1

2. Validate Your ZooKeeper Cluster Health #

Kafka relies entirely on a healthy ZooKeeper cluster, so let’s confirm that first:

• Check myid files: Each ZooKeeper node needs a myid file in its dataDir directory with a value matching its server.X number:
  • Machine 1 (server.1): Run cat /path/to/zookeeper/data/myid — it should return 1
  • Machine 2 (server.2): Same command should return 2
    If this file is missing or has the wrong value, ZooKeeper won’t join the cluster.
• Verify port connectivity: Ensure all required ports are open between the two machines (no firewall/iptables blocks):
  • 2181 (ZooKeeper client port)
  • 2888 (ZooKeeper peer communication)
  • 3888 (ZooKeeper election port)
    Test with nc -zv 10.0.229.48 2181 or telnet 10.0.215.22 2888 from each machine.
• Check ZooKeeper status: Run zkServer.sh status on both machines. One should show Leader, the other Follower. If either says "Error contacting service", start it with zkServer.sh start and check logs in zookeeper.out.

3. Check Kafka Broker Startup & Logs #

Once ZooKeeper is healthy, confirm your Kafka brokers are registering correctly:

• Start order first: Always start both ZooKeeper nodes before starting Kafka brokers.
• Inspect Kafka logs: Check the server.log file in your Kafka logs directory for red flags:
  • Connection to node -1 could not be established: Means Kafka can’t reach ZooKeeper — double-check the zookeeper.connect string and ZooKeeper status.
  • Invalid broker.id: You have duplicate broker IDs (both set to 1) — fix machine 2’s broker.id to 2.
  • Address already in use: Port is occupied — use lsof -i :9094 to find the process and kill it, or change the port.
• Verify broker registration: Use the ZooKeeper client to check if brokers are connected:
  1. Run zkCli.sh -server 10.0.215.22:2181 to connect
  2. Execute ls /brokers/ids — you should see [1,2] if both brokers are registered.

4. Test Cluster Functionality #

Once all the above checks pass, validate end-to-end functionality:

• Create a test topic with replication across both brokers:

  kafka-topics.sh --create --topic test_cluster --bootstrap-server 10.0.215.22:9094,10.0.229.48:9095 --partitions 2 --replication-factor 2
  

• Describe the topic to confirm partition distribution:

  kafka-topics.sh --describe --topic test_cluster --bootstrap-server 10.0.215.22:9094
  

• Send test messages with a producer:

  kafka-console-producer.sh --topic test_cluster --bootstrap-server 10.0.215.22:9094
  

• Receive messages from the other broker to confirm replication:

  kafka-console-consumer.sh --topic test_cluster --bootstrap-server 10.0.229.48:9095 --from-beginning
  

If you hit errors at any step, share the exact log lines or error messages, and we can narrow it down further.

内容的提问来源于stack exchange，提问作者Abhishek Mittal