Minikube空间充足但Elasticsearch Pod提示设备无剩余空间问题排查求助

Hey there, let's break down this tricky issue where your Elasticsearch pods are throwing a "no space left on device" error, even though your Minikube node's disk and inodes look totally fine. Here are the key angles to investigate and fix this:

First, Get More Context from Kubelet Logs #

The pod event only tells you the end result—let's look at the kubelet's detailed logs to see exactly what's failing. SSH into your Minikube node and run:

journalctl -u kubelet -f

Then delete one of the failing Elasticsearch pods and watch the logs. You might see a more specific error (like permission issues instead of actual space constraints) that the pod event didn't surface.

Check Temporary Filesystem Space #

Even though /var/lib/kubelet has plenty of space, kubelet often uses temporary directories like /tmp or /run (which are usually tmpfs volumes with limited size) when setting up projected volumes like kube-api-access. Check their usage:

minikube ssh
df -h /tmp /run
df -ih /tmp /run

If either of these is full, that's likely the culprit. To fix it, restart Minikube with a larger tmpfs allocation:

minikube delete
minikube start --memory 8192 --cpus 4 --disk-size 50000mb --extra-config=kubelet.tmpfs-size=2G

Clean Up Container Runtime Cache #

Over time, unused container images can bloat the filesystem (even if df doesn't show it immediately). Clear the containerd cache in Minikube:

minikube ssh
# Prune unused images
crictl rmi --prune
# Delete stale pod directories (safe if pods are already failing)
rm -rf /var/lib/kubelet/pods/*
# Restart kubelet to pick up changes
systemctl restart kubelet

Then delete your Elasticsearch pods to force them to recreate:

kubectl delete pods -l app=elasticsearch

Verify Security Context Permissions #

The kube-api-access projected volume is owned by root with strict permissions. If your Elasticsearch pods are running as a non-root user (common in Elastic's helm charts), they might not have write access to this volume.

Check your values.yaml for security context settings, and temporarily switch to running as root to test:

securityContext:
  runAsUser: 0
  runAsGroup: 0
  fsGroup: 0

Upgrade your Helm release with this change:

helm upgrade elasticsearch elastic/elasticsearch -f values.yaml

Rule Out SELinux Restrictions #

Minikube's node might have SELinux enforcing policies that block kubelet from writing to the projected volume. Check SELinux status:

minikube ssh
getenforce

If it returns Enforcing, temporarily disable it to test:

setenforce 0

Then restart your Elasticsearch pods. If this fixes the issue, you'll need to adjust SELinux policies or configure your Helm chart to work with SELinux (e.g., adding seLinuxOptions to the pod security context).

内容的提问来源于stack exchange，提问作者Mike Dewar