Azure环境下通过公网IP连接Logstash与Kafka的故障求助

Hey there, I’ve gone through your problem details and the configurations you’ve tried—let’s get to the bottom of why external Logstash can’t reach your Kafka instance, even though it works fine on the same subnet.

The Core Issue: Kafka’s advertised.listeners #

Kafka uses the advertised.listeners setting to tell clients (like Logstash) what address to use to communicate with the broker. When you connect from outside the subnet, Kafka needs to return a public IP that your external client can reach—but your earlier configs either sent an internal IP (which external clients can’t access) or caused leader election issues because of mismatched broker registration.

Step-by-Step Fixes #

1. Update Kafka’s server.properties for Dual Listeners

We’ll set up separate listeners for internal (subnet) and external (public) clients so both can connect properly. Replace placeholders with your actual public IP and internal IP:

# Listen on all network interfaces for both internal and external traffic
listeners=PLAINTEXT://0.0.0.0:9092,PLAINTEXT_EXTERNAL://0.0.0.0:9094

# Tell internal clients to use the private IP, external clients to use the public IP
advertised.listeners=PLAINTEXT://10.10.100.4:9092,PLAINTEXT_EXTERNAL://<YOUR_PUBLIC_IP>:9094

# Map listener names to security protocols (both are plaintext here)
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_EXTERNAL:PLAINTEXT

2. Secure the Azure Network

• Update Network Security Group (NSG): Add an inbound rule allowing traffic on port 9094 from your external Logstash’s IP (use 0.0.0.0/0 for testing, but restrict it later for security).
• Verify VM Firewall: Ensure the OS firewall on your Kafka VM allows inbound traffic on 9094 (you already confirmed 9092 works, but 9094 needs its own rule).

3. Clean Up ZooKeeper & Restart Services

When you change advertised.listeners, old broker registration data in ZooKeeper can cause leader issues:

1. Stop Kafka and ZooKeeper services.
2. Connect to ZooKeeper and delete old broker data:

   zkCli.sh
   rmr /brokers
   exit
   

3. Restart ZooKeeper first, then Kafka.

4. Update Logstash Output Config

Point Logstash to the public IP and external listener port:

kafka {
  bootstrap_servers => "<YOUR_PUBLIC_IP>:9094"
  codec => json
  topic_id => "tagger"
}

Why Your Previous Configs Failed #

• Internal IP only in advertised.listeners: External clients got the private IP, which they couldn’t reach—hence the retry loop errors.
• Public IP only in advertised.listeners: Your internal broker communications (and leader election) tried using the public IP, which might have been blocked or unreachable from the VM itself, leading to LEADER_NOT_AVAILABLE errors.

Additional Troubleshooting Checks #

• Check Kafka Logs: Look for lines starting with Registered broker in server.log to confirm the advertised listeners are correctly registered in ZooKeeper.
• Verify Topic Health: Run this command from your external machine (or the Kafka VM) to check if the topic has a valid leader:

  kafka-topics.sh --describe --topic tagger --bootstrap-server <YOUR_PUBLIC_IP>:9094
  

• Test External Port Connectivity: Use nc -zv <YOUR_PUBLIC_IP> 9094 to confirm the port is reachable from your external Logstash environment.

内容的提问来源于stack exchange，提问作者pm1391