Hey there, let's work through this connection timeout issue step by step. Since your Producer can connect to the GCE Kafka instance just fine, we can focus on narrowing down why the Consumer can't reach it. Here's a structured approach to debugging:

1. First, Verify Basic Network Reachability from GKE to Kafka #

The timeout error strongly suggests a network-level block. Let's confirm if GKE nodes can even reach the Kafka instance's IP and port:

• Spin up a temporary busybox pod in your GKE cluster to test connectivity:

  kubectl run -it --rm busybox --image=busybox:1.36 -- nc -zv 10.128.0.9 9092
  

• If this command times out, the problem is definitely in the network setup. Check these next:
  • GCE Firewall Rules: Ensure your Kafka instance's firewall allows incoming TCP traffic on port 9092 from your GKE cluster's node IP range. You can get the GKE cluster's CIDR with:

    gcloud container clusters describe YOUR_CLUSTER_NAME --zone YOUR_ZONE | grep -A 5 "clusterIpv4Cidr"
    

    Add an inbound firewall rule for this CIDR targeting port 9092.
  • VPC Alignment: Confirm your GKE cluster and GCE Kafka instance are in the same VPC. If they're in different VPCs, you'll need to set up VPC peering to enable communication between them.

2. Check Kafka Listener Configuration #

Kafka's advertised listeners determine what address it tells clients to connect to. If this is misconfigured, even if the network is open, clients might try to connect to an unreachable address:

• SSH into your GCE Kafka instance and check these settings in server.properties:

  cat /opt/kafka/config/server.properties | grep -E "listeners|advertised.listeners"
  

  Ensure advertised.listeners is set to the public/private IP that GKE nodes can access (not localhost or an internal IP only reachable within the GCE instance's local network).
• Verify Kafka is actually listening on the correct port:

  ss -tulpn | grep 9092
  

  Look for output showing Kafka listening on 0.0.0.0:9092 (all interfaces) or the specific IP that GKE can reach.

3. Validate Consumer Deployment Configuration #

Double-check that your Consumer pod is using the right connection settings:

• Confirm the bootstrap.servers value in your Consumer's configuration is exactly 10.128.0.9:9092 (no typos in IP or port).
• Compare your Consumer Deployment YAML with the Producer's. Are there any differences in network-related environment variables (like proxies) or pod settings that might block outgoing traffic?

4. Rule Out GKE Network Policies #

If you've enabled Network Policies in your GKE cluster, they might be blocking outgoing traffic from the Consumer pod to the Kafka instance:

• List all Network Policies in your Consumer's namespace:

  kubectl get networkpolicies -n YOUR_CONSUMER_NAMESPACE
  

• Temporarily delete any restrictive policies (make sure to back them up first) to test if connectivity is restored. If it works, you'll need to add a rule allowing the Consumer pod to access 10.128.0.9:9092.

5. Confirm Kafka Instance Health #

While the Producer works, it's worth checking if Kafka is running smoothly and not dropping connections:

• Check the Kafka service status on the GCE instance:

  systemctl status kafka
  

• Look through Kafka's server logs for any hints of connection issues:

  tail -n 50 /var/log/kafka/server.log
  

  Look for entries about rejected connections or listener failures, though timeout errors usually point to network issues rather than Kafka internal problems.

Start with the network reachability test—it's the most likely cause here. Let me know what you find, and we can dive deeper into the specific issue!

内容的提问来源于stack exchange，提问作者Mitesh Gangaramani