关于Kubernetes Pod内存限制、请求路由及生命周期的技术咨询

Great questions—let’s break each one down with practical, real-world details about how Kubernetes behaves in these scenarios:

1. Pod termination, incoming requests, and load balancing behavior #

When a Pod hits its memory limit and gets terminated (OOMKilled), Kubernetes follows a structured shutdown process to minimize request loss:

• First, the kubelet marks the Pod as Terminating and sends a TERM signal to the main container(s). At the same time, the Kubernetes Endpoints controller removes this Pod from the associated Service’s endpoint list.
• Most cluster load balancers (whether cloud-managed LoadBalancer, NodePort, or kube-proxy for ClusterIP Services) sync this endpoint change quickly. Once the Pod is removed from the list, new incoming requests won’t be routed to it anymore—they’ll automatically go to other available Pods in the Deployment.
• There’s a tiny edge case window where the LB might still have stale endpoint data, but this is usually negligible. For the terminating Pod itself: if your app handles the TERM signal properly, it should finish processing existing requests and stop accepting new ones during the grace period. If your app ignores the signal, the kubelet will wait for the default terminationGracePeriodSeconds (30 seconds) before sending a SIGKILL to force termination, which could drop any in-flight requests at that point.

2. All Pods hit memory limits, slow Pod creation, and LB behavior #

If every Pod in your Deployment gets OOMKilled at the same time, Kubernetes will start spinning up new Pods—but if that takes minutes (e.g., slow image pulls, node resource contention), here’s what you’ll see:

• The Service’s endpoint list will be empty because there are no ready Pods to route to.
• When the load balancer tries to send requests, it has no backends available. This means requests will be dropped or return a 5xx error (like 503 Service Unavailable). Kubernetes doesn’t queue requests automatically—you’d need to handle this at the client side (with retry logic) or LB level (if your LB supports request queuing).
• To avoid this stressful scenario entirely, I’d recommend setting up a Horizontal Pod Autoscaler (HPA) that scales your Deployment based on memory usage before Pods hit the limit. For example, configure HPA to scale out when memory usage hits 70% of your request/limit value. This adds capacity proactively instead of waiting for OOM kills.

3. Configuring Pod lifecycle behavior #

Absolutely—Kubernetes gives you several flexible tools to control the Pod lifecycle:

• Graceful termination: Use the terminationGracePeriodSeconds field in your Pod spec to set how long Kubernetes waits after sending the TERM signal before force-killing the Pod. This gives your app time to wrap up in-flight requests.
• Lifecycle hooks: Define postStart (runs right after the container starts) and preStop (runs just before termination) hooks. For example, a preStop hook could call an API to notify your service mesh to stop routing traffic, or trigger a custom graceful shutdown script in your app.
• Readiness probes: These checks ensure a Pod is only added to the Service’s endpoint list once it’s fully ready to handle requests. You can check a HTTP endpoint, a TCP port, or run a command—this prevents traffic from being sent to Pods that are still warming up.
• Liveness probes: These detect if a Pod is unresponsive and trigger a restart if needed. Useful for recovering from app crashes or deadlocks.
• Init containers: Run one or more containers before your main container starts to handle setup tasks (like waiting for a database to be available, pulling configuration files, or running migrations).

内容的提问来源于stack exchange，提问作者Rams