Kubernetes中gRPC服务TCP访问异常的调试步骤咨询

Let’s work through the next steps to diagnose your inaccessible TCP/gRPC service, building on the checks you’ve already completed (pod readiness, endpoint association, node IO verification). First, let’s fix that telnet installation snag, then dive into targeted debugging:

Fixing Telnet Installation (or Using an Alternative) #

Minimal container images often skip common tools like telnet. Try these alternatives depending on your base image:

• Debian/Ubuntu-based images:

  apt update && apt install -y telnet
  

• Alpine-based images:

  apk add --no-cache telnet
  

• If no package manager is available: Use netcat (nc), which is pre-installed in many minimal images. Test connectivity with:

  nc -zv <target-ip> <port>
  

  The -z flag checks for open ports without sending data, and -v enables verbose output.

Next Debugging Steps #

1. Verify Local Port Listening in the Pod #

First, confirm your gRPC service is actually listening on the expected port and binding to a non-loopback address (0.0.0.0, not 127.0.0.1). Run this inside the pod:

# For modern systems
ss -tulpn | grep <your-service-port>
# Or if netstat is available
netstat -tulpn | grep <your-service-port>

If the output shows 127.0.0.1:<port>, your service is only accessible from inside the pod itself—update its configuration to bind to 0.0.0.0.

2. Test Connectivity from Another Pod in the Cluster #

Launch a temporary test pod to isolate cluster-internal connectivity issues:

kubectl run -it --rm --image=busybox:1.36 test-pod -- /bin/sh

From this pod, test access to your service using:

• Service ClusterIP: nc -zv <service-cluster-ip> <service-port>
• Pod IP directly: nc -zv <pod-ip> <pod-port>
  If connecting to the Pod IP works but the Service IP doesn’t, double-check your Service’s targetPort matches the pod’s exposed port, and confirm the Service selector labels exactly match the pod’s labels (even a typo here can break things, even if endpoints look correct at first glance).

3. Check for NetworkPolicy Restrictions #

If your cluster uses NetworkPolicies, they might be blocking incoming TCP traffic to your service. List all policies with:

kubectl get networkpolicies --all-namespaces

Look for policies targeting your service’s namespace or pod labels. To rule this out temporarily, create a permissive policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-to-my-service
  namespace: <your-namespace>
spec:
  podSelector:
    matchLabels:
      <your-pod-label-key>: <your-pod-label-value>
  ingress:
  - {}

Apply it with kubectl apply -f <policy-file.yaml> and retest connectivity.

4. Node-Level Packet Capture with tcpdump #

Log into the node where your pod is running, then capture traffic to/from your service port:

# Capture traffic to the Service NodePort (if using NodePort type)
tcpdump -i any port <node-port> -v
# Or capture traffic to the pod's port
tcpdump -i any port <pod-port> -v

While the capture is running, attempt to connect from outside the cluster. If you see incoming packets but no responses, the issue is likely in the CNI plugin (e.g., Calico, Flannel) routing traffic to the pod. Check the CNI agent logs on the node for errors.

5. Debug gRPC-Specific Behavior #

If TCP connects but gRPC calls fail, use grpcurl (if you can install it) to validate the service:

# Install grpcurl in a Debian/Ubuntu pod
apt update && apt install -y grpcurl
# List available gRPC services
grpcurl -plaintext <target-ip>:<port> list

If you can’t install grpcurl, check your service’s logs for gRPC-specific errors—common issues include misconfigured TLS (if enabled) or invalid service method names.

6. Validate DNS Resolution (If Using Service Names) #

If you’re accessing the service via its name (not IP), test DNS resolution inside your pod:

nslookup <your-service-name>

If it fails, check the CoreDNS pods in the kube-system namespace:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system <coredns-pod-name>

Look for errors related to DNS resolution or namespace issues.

7. Check kube-proxy Status #

kube-proxy manages the network rules that route traffic to Services. On your node, check its logs:

# For systemd-based nodes
journalctl -u kube-proxy -f
# Or if kube-proxy runs as a pod
kubectl logs -n kube-system <kube-proxy-pod-name> -f

You can also verify the iptables rules for your service:

iptables-save | grep <service-cluster-ip>

Missing or incorrect rules here can prevent traffic from reaching the service.

内容的提问来源于stack exchange，提问作者nmiculinic