如何追踪Kubernetes的OOMKilled终止信息？解决日志缺失问题

Hey there! Let's break down why you're not seeing the OOMKilled details where you expect them, and walk through how to track them down.

First, Understand Why /dev/termination-log is Empty #

When a container gets OOMKilled by the kernel, it’s sent a SIGKILL signal instantly—there’s no time for the container process to write to /dev/termination-log before it’s terminated. This empty file isn’t a bug, it’s totally normal behavior for OOM events.

Here’s How to Actually Find the OOMKilled Details #

Let’s go through the most reliable places to track down these events:

• Check the Pod’s Events with kubectl describe
  This is the quickest way to confirm an OOMKilled event. Run:

  kubectl describe pod <your-pod-name>
  

  Look in the Events section—you should see a warning entry like:

  Warning OOMKilled 5m kubelet Container was terminated due to OOMKilled

• Inspect the Kubelet Logs
  The kubelet is responsible for monitoring container resource usage and triggering OOM kills when limits are hit. To view its logs (on systemd-based nodes):

  journalctl -u kubelet -f
  

  You’ll find detailed entries about the OOM event, including which container was killed, the memory limit that was exceeded, and precise timestamps.

• Check Container Runtime Metadata
  If you’re using containerd or another runtime, you can pull termination details directly from the container’s metadata. First get the container ID with:

  kubectl get pod <your-pod-name> -o jsonpath='{.status.containerStatuses[0].containerID}'
  

  Then use your runtime’s tool (like crictl for containerd) to inspect it:

  crictl inspect <container-id>
  

  Look for the status.state.terminated section—you’ll see reason: "OOMKilled" along with any associated context about the memory exhaustion.

• Why terminationMessagePolicy=FallbackToLogsOnError Isn’t Showing Anything
  This setting falls back to the container’s logs only if the termination log is empty—but if the OOM happened before the container had a chance to write any logs, there’s nothing to pull. Additionally, if your logs are being rotated or collected off-node quickly, the relevant entries might already be gone by the time you check.

Quick Reality Check #

Make sure your container has a memory limit set (resources.limits.memory in your Pod spec). Without a limit, the node’s kernel OOM killer will handle memory exhaustion instead of kubelet, and the Pod might show as Evicted or NodeLost rather than OOMKilled—and those events will show up in node logs instead of container-level logs.

内容的提问来源于stack exchange，提问作者Matheus Neder