使用kafka-console-producer发送消息遇Topic超时错误求助

Hey Mina, let's work through this Kafka producer issue step by step. The error TimeoutException: Topic testTopic not present in metadata after 60000 ms is a common one, and it usually stems from a few key issues. Here's how to diagnose and fix it:

1. Verify the testTopic actually exists #

First, confirm the topic is created on your Kafka cluster. Run this command to list all topics:

kafka-topics --list --bootstrap-server confluent-kafka-0-service.ms-kafka-internal.svc:9092 --command-config ~/etc/kafka/client_security.properties

If testTopic doesn't show up, create it with:

kafka-topics --create --topic testTopic --bootstrap-server confluent-kafka-0-service.ms-kafka-internal.svc:9092 --partitions 1 --replication-factor 1 --command-config ~/etc/kafka/client_security.properties

(Adjust partitions/replication factor based on your cluster's actual setup)

2. Check network connectivity to the Kafka broker #

Since your broker address uses a Kubernetes Service suffix (svc), ensure your producer machine can reach the broker's port:

• Test basic connectivity with:

  nc -zv confluent-kafka-0-service.ms-kafka-internal.svc 9092
  

  Or if nc isn't available:

  telnet confluent-kafka-0-service.ms-kafka-internal.svc 9092
  

• If you're in a Kubernetes environment, confirm:
  • The producer pod is in the same namespace as the Kafka Service (ms-kafka-internal)
  • No NetworkPolicies are blocking traffic to the Kafka Service

3. Validate your security configuration #

Your producer/consumer uses a security properties file—make sure it's correctly configured:

• Open ~/etc/kafka/client_security.properties and check for critical settings like:

  security.protocol=SASL_SSL  # Or SSL, depending on your cluster setup
  sasl.mechanism=PLAIN        # Adjust if using a different mechanism like SCRAM-SHA-256
  ssl.truststore.location=/path/to/truststore.jks
  ssl.truststore.password=your-truststore-password
  

• Double-check that the file path in your producer command is correct (you used ~/etc/kafka/client_security.properties, but your consumer command uses ~/etc/bmw/kafka/client_security.properties—this might be a typo!)
• Test with explicit security properties in the command to rule out config file issues:

  /usr/bin/kafka-console-producer --broker-list confluent-kafka-0-service.ms-kafka-internal.svc:9092 --topic testTopic \
    --producer-property security.protocol=SASL_SSL \
    --producer-property sasl.mechanism=PLAIN \
    --producer-property ssl.truststore.location=/path/to/truststore.jks \
    --producer-property ssl.truststore.password=yourpassword
  

4. Check Kafka broker health and configuration #

• Ensure the Kafka broker pod (confluent-kafka-0) is running and healthy. Check its logs for startup errors or connectivity issues:

  # If using Kubernetes
  kubectl logs confluent-kafka-0 -n ms-kafka-internal
  

• Verify the broker's listeners and advertised.listeners settings. For a Kubernetes setup, the advertised.listeners should point to the Service address so producers can reach it:

  listeners=PLAINTEXT://0.0.0.0:9092
  advertised.listeners=PLAINTEXT://confluent-kafka-0-service.ms-kafka-internal.svc:9092
  

5. Fix the consumer command path discrepancy #

Notice your consumer command uses ~/etc/bmw/kafka/client_security.properties while the producer uses ~/etc/kafka/client_security.properties. Make sure both point to the correct, existing configuration file to avoid authentication issues.

If you've gone through all these steps and still hit the error, feel free to share more details like your broker's full configuration or security settings, and we can dig deeper.

内容的提问来源于stack exchange，提问作者MinaSri