Kafka生产者与消费者报错求助：LeaderNotAvailableException问题

Hey there, let's break down and fix your Kafka issues step by step!

Your Actions & Error Details #

First, let's recap what you ran and the errors you encountered:

Console 1 (Producer Commands)

export PATH=$PATH:/usr/hdp/current/kafka-broker/bin 
kafka-topics.sh --create --zookeeper ip-172-31-20-58.ec2.internal:2181 --replication-factor 1 --partitions 1 --topic testuday1234 
kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234

Console 2 (Consumer Command)

export PATH=$PATH:/usr/hdp/current/kafka-broker/bin 
kafka-console-consumer.sh --zookeeper localhost:2181 --topic testuday123 --from-beginning

Producer Error Logs

[udaychitukula6587@ip-172-31-38-183 ~]$ kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234 hi 
[2018-05-28 15:27:36,761] WARN Error while fetching metadata [{TopicMetadata for topic testuday1234 -> No partition metadata for topic testuday1234 due to kafka.common.LeaderNotAvailableException}] for topic [testuday1234]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) 
[2018-05-28 15:27:36,774] WARN Error while fetching metadata [{TopicMetadata for topic testuday1234 -> No partition metadata for topic testuday1234 due to kafka.common.LeaderNotAvailableException}] for topic [testuday1234]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)

Consumer Abnormal Output

[udaychitukula6587@ip-172-31-38-183 ~]$ kafka-console-consumer.sh --zookeeper localhost:2181 --topic testuday123 --from-beginning 
{metadata.broker.list=ip-172-31-20-58.ec2.internal:6667,ip-172-31-53-48.ec2.internal:6667,ip-172-31-60-179.ec2.internal:6667, request.timeout.ms=30000, client.id=console-consumer-63526, security.protocol=PLAINTEXT} 
{metadata.broker.list=ip-172-31-20-58.ec2.internal:6667,ip-172-31-53-48.ec2.internal:6667,ip-172-31-60-179.ec2.internal:6667, request.timeout.ms=30000, client.id=console-consumer-63526, security.protocol=PLAINTEXT}

Step-by-Step Fixes #

Let's tackle these issues one by one:

1. Resolve Producer's LeaderNotAvailableException

This error typically pops up when you try producing to a topic right after creating it—Kafka needs a few seconds to elect a leader for the topic's partition. Here's how to fix it:

• First, confirm if the topic was actually created successfully:

  kafka-topics.sh --list --zookeeper ip-172-31-20-58.ec2.internal:2181
  

• If testuday1234 isn't in the list, re-create the topic, then wait 30 seconds before starting the producer (give Kafka enough time to finish leader election):

  kafka-topics.sh --create --zookeeper ip-172-31-20-58.ec2.internal:2181 --replication-factor 1 --partitions 1 --topic testuday1234
  # Wait 30 seconds
  kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234
  

2. Fix Consumer's Topic Name Typo

A critical mistake here: your consumer command uses --topic testuday123, but you created the topic as testuday1234 (missing a 4 at the end!). Correct this with one of these commands:

• For older Kafka versions (using ZooKeeper-based consumer):

  kafka-console-consumer.sh --zookeeper ip-172-31-20-58.ec2.internal:2181 --topic testuday1234 --from-beginning
  

• For newer Kafka versions (recommended, since ZooKeeper-based consumers are deprecated):

  kafka-console-consumer.sh --bootstrap-server ip-172-31-20-58.ec2.internal:6667 --topic testuday1234 --from-beginning
  

3. Additional Checks for Stability

• Verify Kafka Broker is running: Use jps to check if the Kafka process is active, or review the Broker logs in your HDP cluster.
• Test network connectivity: Ensure your producer/consumer machines can reach the Broker's 6667 port and ZooKeeper's 2181 port (e.g., run telnet ip-172-31-20-58.ec2.internal 6667 to test).
• Check HDP Kafka configs: Make sure listeners and advertised.listeners are correctly configured so Brokers are accessible from external machines.

内容的提问来源于stack exchange，提问作者user2381482