You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

EKS部署cAdvisor报无法配置OOM检测源问题排查

问题描述
  • 部署环境:AWS EKS Kubernetes集群
  • 部署对象:cAdvisor DaemonSet,部署使用的完整清单如下:
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cadvisor
  namespace: kube-monitoring
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cadvisor
rules:
  - apiGroups: ['policy']
    resources: ['podsecuritypolicies']
    verbs:     ['use']
    resourceNames:
    - cadvisor
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cadvisor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cadvisor
subjects:
- kind: ServiceAccount
  name: cadvisor
  namespace: kube-monitoring
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: kube-monitoring
spec:
  selector:
    matchLabels:
      name: cadvisor
  template:
    metadata:
      labels:
        name: cadvisor
    spec:
      serviceAccountName: cadvisor
      containers:
      - name: cadvisor
        image: google/cadvisor:latest
        resources:
          requests:
            memory: 400Mi
            cpu: 400m
          limits:
            memory: 2000Mi
            cpu: 800m
        ports:
          - name: http
            containerPort: 8080
            protocol: TCP
        volumeMounts:
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: true
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: disk
          mountPath: /dev/disk
          readOnly: true
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      volumes:
      - name: rootfs
        hostPath:
          path: /
      - name: var-run
        hostPath:
          path: /var/run
      - name: sys
        hostPath:
          path: /sys
      - name: docker
        hostPath:
          path: /var/lib/docker
      - name: disk
        hostPath:
          path: /dev/disk
---
  • 异常现象:cAdvisor容器持续输出如下告警日志
W0608 16:00:47.238042       1 manager.go:349] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
  • 正常表现:可正常访问http://localhost:8080/containers/路径下的cAdvisor UI页面,需要定位配置缺陷。
问题根因

该告警由配置缺失直接导致:cAdvisor容器未挂载宿主机的/dev/kmsg设备。
cAdvisor的OOM事件检测逻辑依赖读取内核暴露的/dev/kmsg字符设备,从中解析内核输出的OOM Kill事件记录。现有配置的挂载列表仅覆盖了根文件系统、运行时目录、sysfs、Docker数据目录、磁盘设备路径,遗漏了/dev/kmsg设备,容器内不存在该设备文件,因此cAdvisor初始化时无法启动OOM检测模块,自动禁用该功能并输出告警。
该缺失不会影响cAdvisor核心的容器资源指标采集、Web UI服务能力,因此基础访问功能完全正常,仅OOM事件检测特性不可用。

修复方法

修改cAdvisor DaemonSet配置,补充/dev/kmsg设备的只读挂载:

  1. 在spec.template.spec.volumes段新增卷定义:
- name: kmsg
  hostPath:
    path: /dev/kmsg
  1. 在cAdvisor容器的volumeMounts段新增挂载规则:
- name: kmsg
  mountPath: /dev/kmsg
  readOnly: true

配置更新后DaemonSet会滚动重建所有cAdvisor Pod,新Pod启动后将不再输出该告警,OOM事件检测功能恢复正常。EKS默认的Amazon Linux 2、Amazon Linux 2023节点镜像默认自带/dev/kmsg设备,无需在节点侧做额外调整。


内容的提问来源于stack exchange,提问作者rp346

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.09.02 06:46:04