GKE沙箱集群内部负载均衡器实例组丢失问题求助

Hey there, let's break down this issue step by step—your hunch about the Pod scaling to 0 is definitely a strong lead, but we should also cross-check with the preemptive nodes to be thorough. Here's how to diagnose and fix this:

First, Verify the LB-Service Relationship #

• Grab details of your internal LoadBalancer Service with kubectl describe service <your-service-name>. Look for the LoadBalancer Ingress field to get the associated GCP backend service name.
• Use gcloud compute backend-services describe <backend-service-name> to check which instance groups are linked. Do this right after scaling Pods to 0, and again when the issue occurs—this will confirm if the instance group is being removed when Pods hit 0 replicas.

Check GKE's Behavior When Pods Scale to 0 #

• When you scale a Deployment to 0, the corresponding Endpoints resource for the Service will become empty. Run kubectl get endpoints <your-service-name> to confirm this.
• Head over to Cloud Logging and search for logs related to your backend service and Service name. Look for entries mentioning "removing instance group" or similar around the time you scale Pods down—this will tell you if GKE is intentionally detaching the instance group when there are no endpoints.

Rule Out Preemptive Node Interference #

• Preemptive nodes can be terminated at any time, which might affect the instance group linked to your LB. Check if node termination times line up with your endpoint outages using:

  gcloud compute instances list --filter="status=TERMINATED" --sort-by=terminationTimestamp
  

• If your Service uses externalTrafficPolicy: Local, terminated nodes would be removed from the instance group—but if Pods were already scaled to 0, this could leave the backend service with no linked groups entirely.

Potential Fixes to Try #

• Avoid scaling to 0 replicas entirely: Instead of scaling down to 0, keep a single "placeholder" Pod running. This ensures the Endpoints resource stays populated, and GKE won't remove the instance group from the backend service. Adjust your off-hours scaling script to use --replicas=1 instead of 0.
• Lock in the backend service instance group: If you're comfortable modifying the GCP backend service directly, you can manually add the instance group (used by your GKE node pool) and configure it to stay linked even if there are no active Pods. Note that this might require adjusting health checks to avoid false negatives when only the placeholder Pod is running.
• Check your GKE version: Older GKE versions had bugs where scaling Pods to 0 would incorrectly unlink instance groups from internal LBs. Upgrading to the latest stable minor version might resolve the issue.
• Use a PodDisruptionBudget: Set a PDB with minAvailable: 1 for your Deployments—this prevents accidental scaling to 0 (though you'd need to adjust your script to respect this, or use it as a safeguard against unintended changes).

Final Notes #

Start by confirming whether scaling to 0 triggers the instance group removal (via logs and immediate checks after scaling). The placeholder Pod trick is usually the quickest fix to test if this resolves the weekly outages. If preemptive nodes are also contributing, combining the placeholder with a small number of non-preemptive nodes in your pool might add extra stability.

内容的提问来源于stack exchange，提问作者caarlos0