启用Kerberos后Kafka控制台生产者报security-protocol未识别错误求助

Hey there, let's work through this Kafka Kerberos issue you're hitting! That error about security-protocol not being a recognized option is super common with the Kafka version bundled in HDP 3.1.0—here's why and how to fix it:

The Root Cause #

HDP 3.1.0 ships with Kafka 1.1.0, and in this older version, the console producer/consumer scripts don't support passing security-related flags like --security-protocol directly on the command line. Instead, you need to define these settings in a properties file and reference it in your command.

Step 1: Create a Producer Security Config File #

First, make a properties file (let's call it kafka-producer-sasl.properties) with the following content:

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka

• sasl.kerberos.service.name must match the service name configured for your Kafka brokers (this is almost always kafka in HDP setups).

Step 2: Run the Producer with the Config File #

Replace your original producer command with this one, pointing to the config file you just created:

/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list HOSTNAME:6667 --topic test_new_topic --producer.config /path/to/kafka-producer-sasl.properties

Step 3: Set Up the Consumer (Same Logic!) #

For the consumer, create a similar config file (kafka-consumer-sasl.properties):

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
group.id=test-consumer-group

Then run the consumer command:

/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-consumer.sh --bootstrap-server HOSTNAME:6667 --topic test_new_topic --from-beginning --consumer.config /path/to/kafka-consumer-sasl.properties

Quick Pre-Flight Check #

Before running these commands, make sure you've got a valid Kerberos ticket for the principal that has access to the Kafka topic. Use kinit if you haven't already:

kinit -kt /path/to/your-service-account.keytab your-principal@YOUR-REALM.COM

That should get your console producer and consumer working with Kerberized Kafka in HDP 3.1.0!

内容的提问来源于stack exchange，提问作者pujara