搭建带SSL加密与SASL/Kerberos认证的3节点Kafka集群时发送消息报错

Let's tackle this metadata update timeout issue you're facing when sending messages to the test topic. This error usually points to a failure in broker-client authentication or network connectivity that blocks the client from fetching cluster metadata. Here's a step-by-step breakdown of the most likely fixes:

1. Verify Broker Kerberos Configuration & Keytab Validity #

First, ensure each broker's server.properties has correct Kerberos settings and the keytab is properly deployed:

• Confirm these parameters are set correctly (replace <broker-hostname> with your actual broker names like broker1):

  sasl.kerberos.service.name=kafka
  security.inter.broker.protocol=SASL_SSL
  sasl.enabled.mechanisms=GSSAPI
  listeners=SASL_SSL://<broker-hostname>:9093
  advertised.listeners=SASL_SSL://<broker-hostname>:9093
  

  The advertised.listeners must match the hostname in your Kerberos principal (kafka/brokerX@EXAMPLE.COM) and be resolvable by clients.
• Validate the keytab on each broker:
  • Run klist -kt /path/to/kafka-server.keytab to confirm it contains the correct principal (kafka/brokerX@EXAMPLE.COM).
  • Lock down keytab permissions: chown kafka:kafka /path/to/kafka-server.keytab and chmod 600 /path/to/kafka-server.keytab—Kerberos rejects keytabs with open, insecure permissions.

2. Fix Client Producer Configuration & Kerberos Permissions #

Your message producer needs valid Kerberos credentials and proper SSL settings to connect:

• Create a JAAS config file for the producer (e.g., producer-jaas.conf) using a dedicated client principal (you'll need to create this principal in your Kerberos server first):

  KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      keyTab="/path/to/client-producer.keytab"
      principal="kafka-producer@EXAMPLE.COM";
  };
  

• Set the JAAS config before launching the producer:

  export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/producer-jaas.conf"
  

• Ensure your producer.properties includes these SSL/SASL settings:

  security.protocol=SASL_SSL
  sasl.mechanism=GSSAPI
  sasl.kerberos.service.name=kafka
  ssl.truststore.location=/path/to/truststore.jks
  ssl.truststore.password=<your-truststore-password>
  

• Critical: Grant the client principal write access to the test topic using Kafka ACLs:

  kafka-acls.sh --authorizer-properties zookeeper.connect=<zk-host>:2181 \
    --add --allow-principal User:kafka-producer \
    --operation Write --topic test
  

3. Validate Hostname Resolution & Network Connectivity #

Kerberos relies on accurate hostname matching, so resolve any DNS/hosts file issues:

• Check that all brokers and the client can resolve each other's hostnames (e.g., ping broker1 from broker2 and the client should return the correct IP).
• Update /etc/hosts on all nodes if DNS isn't working, adding entries like:

  <broker1-ip> broker1
  <broker2-ip> broker2
  <broker3-ip> broker3
  

• Ensure the port specified in listeners (e.g., 9093) is open between all nodes (use telnet broker1 9093 from the client to test connectivity).

4. Check SSL Certificate Validity #

SSL misconfigurations can also block metadata fetching:

• Verify each broker's keystore and truststore have valid certificates:

  keytool -list -v -keystore /path/to/kafka.keystore.jks
  

  Ensure the certificate's Common Name (CN) matches the broker's hostname and the certificate hasn't expired.
• Confirm the client's truststore contains the CA certificate used to sign the broker's SSL certificates—otherwise, the client will reject the broker's SSL connection.

5. Inspect Logs for Detailed Errors #

If the above steps don't resolve the issue, dig into logs for specific failure details:

• Check broker logs (default path: /var/log/kafka/server.log) for keywords like Kerberos, Authentication, or Metadata. Look for errors like Failed to authenticate client or Could not find leader for topic.
• Enable debug logging for the producer to get granular context:

  export KAFKA_OPTS="$KAFKA_OPTS -Dsun.security.krb5.debug=true -Djavax.net.debug=ssl"
  

  This will print verbose Kerberos and SSL handshake logs, which often pinpoint the exact failure.

内容的提问来源于stack exchange，提问作者Rishi Reddy