EKS部署cAdvisor报无法配置OOM检测源问题排查
问题描述
- 部署环境:AWS EKS Kubernetes集群
- 部署对象:cAdvisor DaemonSet,部署使用的完整清单如下:
--- apiVersion: v1 kind: ServiceAccount metadata: name: cadvisor namespace: kube-monitoring --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cadvisor rules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - cadvisor --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cadvisor roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cadvisor subjects: - kind: ServiceAccount name: cadvisor namespace: kube-monitoring --- apiVersion: apps/v1 kind: DaemonSet metadata: name: cadvisor namespace: kube-monitoring spec: selector: matchLabels: name: cadvisor template: metadata: labels: name: cadvisor spec: serviceAccountName: cadvisor containers: - name: cadvisor image: google/cadvisor:latest resources: requests: memory: 400Mi cpu: 400m limits: memory: 2000Mi cpu: 800m ports: - name: http containerPort: 8080 protocol: TCP volumeMounts: - name: rootfs mountPath: /rootfs readOnly: true - name: var-run mountPath: /var/run readOnly: true - name: sys mountPath: /sys readOnly: true - name: docker mountPath: /var/lib/docker readOnly: true - name: disk mountPath: /dev/disk readOnly: true automountServiceAccountToken: false terminationGracePeriodSeconds: 30 volumes: - name: rootfs hostPath: path: / - name: var-run hostPath: path: /var/run - name: sys hostPath: path: /sys - name: docker hostPath: path: /var/lib/docker - name: disk hostPath: path: /dev/disk ---
- 异常现象:cAdvisor容器持续输出如下告警日志
W0608 16:00:47.238042 1 manager.go:349] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
- 正常表现:可正常访问
http://localhost:8080/containers/路径下的cAdvisor UI页面,需要定位配置缺陷。
问题根因
该告警由配置缺失直接导致:cAdvisor容器未挂载宿主机的/dev/kmsg设备。
cAdvisor的OOM事件检测逻辑依赖读取内核暴露的/dev/kmsg字符设备,从中解析内核输出的OOM Kill事件记录。现有配置的挂载列表仅覆盖了根文件系统、运行时目录、sysfs、Docker数据目录、磁盘设备路径,遗漏了/dev/kmsg设备,容器内不存在该设备文件,因此cAdvisor初始化时无法启动OOM检测模块,自动禁用该功能并输出告警。
该缺失不会影响cAdvisor核心的容器资源指标采集、Web UI服务能力,因此基础访问功能完全正常,仅OOM事件检测特性不可用。
修复方法
修改cAdvisor DaemonSet配置,补充/dev/kmsg设备的只读挂载:
- 在
spec.template.spec.volumes段新增卷定义:
- name: kmsg hostPath: path: /dev/kmsg
- 在cAdvisor容器的
volumeMounts段新增挂载规则:
- name: kmsg mountPath: /dev/kmsg readOnly: true
配置更新后DaemonSet会滚动重建所有cAdvisor Pod,新Pod启动后将不再输出该告警,OOM事件检测功能恢复正常。EKS默认的Amazon Linux 2、Amazon Linux 2023节点镜像默认自带/dev/kmsg设备,无需在节点侧做额外调整。
内容的提问来源于stack exchange,提问作者rp346
相关产品推荐
相关产品推荐

