MicroK8s节点ephemeral-storage不足告警求助：df显示空间充足

问题详情 #

已删除集群内所有对象并重新创建集群，但节点仍提示：The node was low on resource: ephemeral-storage（节点临时存储不足）。执行df命令显示系统分区有充足可用空间，检查microk8s ctr images list未发现镜像冗余。系统分区使用率曾短暂飙升至96%，但后续执行df时该情况未持续。

相关命令输出 #

df -h 输出

light@o-node0:~/lh-orchestrator$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.2G  1.9M  1.2G   1% /run
/dev/sda2        32G   23G  7.6G  75% /
tmpfs           5.8G     0  5.8G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/7b67cbf0dfd60e763e26be4555ed80ec129c14e902eed464e4831e0ae5df3bf8/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/d688bf0e57d05c8165afb32250e6bc216f6072c3b86e96cad26e3caae4c8d856/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/36ec3d840cf8f3c1bd5b4953fa445c71bceb1f0afa5b13fa24855fd17e99bb1e/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/95546ac44384550e3446059da4f7d24cc7a4bfe21d00e67bd924d6a7bd77b650/shm
tmpfs           1.2G  4.0K  1.2G   1% /run/user/1000
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/0b61b66534868b6d588f6ee0d752a416cf5b4849673dde666d1eaf74088e0057/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/0193379ce5b4c60a5c31a402ba0248dd38691d8f4e44952a7bfa2b5db0738149/shm
shm              64M   16K   64M   1% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/1e0381a9c0c430b5ef54f2f7490addb68524220296b55e61b8ecfbdc2c6fbd0c/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/196481a3394e8b66b363fb78bb1824744a090d67af50fb80929c4fc0be78b7e0/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/bb2448e3cecbc98ba64202f175a8c6a21f144bd750eb3e369a0f74e5c6f2df3b/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/1d6f88440c6e5e88656ddcb4441f1700f9b9df2e1598011d01099a9fa4b043e0/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/787853e894de1df24ec204659a3a519a64e6b51776488ee27624cff8baae862b/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/7fe5df9a0b9e96503cac42c7bcfb1a27630ff6534ce075e974e35769697d8bc8/shm

kubectl describe pod 输出

light@o-node0:~$ k describe pod aa114-detect-7b4b986856-c22h6
Name:             aa114-detect-7b4b986856-c22h6
Namespace:        default
Priority:         0
Service Account:  default
Node:             o-node0/192.168.125.74
Start Time:       Thu, 27 Apr 2023 05:43:26 +0000
Labels:           app=aa114-detect
                  pod-template-hash=7b4b986856
Annotations:      cni.projectcalico.org/containerID: 1f7cbc2faac67746999cb3380fddf1a47c0b9748800795a8ea72dfe1feb0a1db
                  cni.projectcalico.org/podIP: 
                  cni.projectcalico.org/podIPs: 
Status:           Failed
Reason:           Evicted
Message:          The node was low on resource: ephemeral-storage. Threshold quantity: 1Gi, available: 0. 
IP:               10.1.234.169
IPs:
  IP:           10.1.234.169
Controlled By:  ReplicaSet/aa114-detect-7b4b986856
Containers:
  aa114-detect:
    Container ID:  
    Image:         registry.dev.mpksoft.ru/lighthouse/lh-detector/detect-runner-shliakhtin.img
    Image ID:      
    Port:          8554/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
    Args:
      -c
      /usr/bin/lh-detector --object-detector.camera_url="$(CAMERA_URL)" $(DETECTOR_EXTRA_ARGS)
    State:          Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was terminated
      Exit Code:    137
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Last State:     Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was deleted.  The container used to be Running
      Exit Code:    137
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  1
    Environment Variables from:
      aa114-pipel-config  ConfigMap  Optional: false
    Environment:          <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jvpf8 (ro)
Conditions:
  Type               Status
  DisruptionTarget   True 
  Initialized        True 
  Ready              False 
  ContainersReady    False 
  PodScheduled       True 
Volumes:
  kube-api-access-jvpf8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

排查与解决步骤 #

1. 检查容器日志占用 #

Kubernetes的临时存储包含容器日志，大量日志会瞬间占用磁盘空间：

• 查看MicroK8s容器日志目录大小：

  du -sh /var/snap/microk8s/common/var/log/pods/
  

• 若日志过大，配置日志轮转：
  修改containerd配置文件/var/snap/microk8s/current/args/containerd-template.toml，添加日志轮转参数：

  [plugins."io.containerd.grpc.v1.cri".containerd.log]
    format = "json"
    max_size = "100Mi"
    max_files = 5
  

  重启MicroK8s生效：microk8s stop && microk8s start

2. 定位瞬时存储占用来源 #

使用工具监控磁盘写入，找到峰值产生的进程或容器：

• 实时监控磁盘IO：

  iostat -x 1
  

• 排查已删除但未释放的文件（这类文件会占用磁盘空间）：

  lsof | grep deleted
  

  找到对应进程后重启即可释放空间。

3. 调整Kubelet驱逐阈值 #

当前节点驱逐阈值为可用空间低于1Gi，可适当调高：

• 编辑MicroK8s的kubelet配置文件/var/snap/microk8s/current/args/kubelet，添加或修改参数：

  --eviction-hard=nodefs.available<2Gi,imagefs.available<1Gi
  --eviction-soft=nodefs.available<3Gi,imagefs.available<2Gi
  --eviction-soft-grace-period=nodefs.available=2m,imagefs.available=2m
  

• 重启MicroK8s使配置生效：microk8s stop && microk8s start

4. 限制EmptyDir卷占用 #

若集群内其他Pod使用了EmptyDir卷，默认会占用节点存储，可添加大小限制：

volumes:
- name: temp-storage
  emptyDir:
    sizeLimit: 500Mi

内容的提问来源于stack exchange，提问作者noname7619