如何在Prometheus告警中正确展示node_filesystem_avail_bytes剩余空间

需要在Prometheus告警中展示剩余磁盘空间，将node_filesystem_avail_bytes指标通过直接除以1024*1024转换为MB，但当前告警规则的转换逻辑存在数值偏差：

• 现有告警规则：

- alert: HostOutOfDiskSpace_test
  expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes > 0 and ON (instance, device, mountpoint) node_filesystem_readonly == 0 and {instance=~"10.22.22.18.*"}
  for: 1s
  labels:
    severity: high
  annotations:
    summary: "Host {{ $labels.instance }} out of memory \n - Device: {{ $labels.device }} \n - Mountpoint - {{ $labels.mountpoint }}"
    description: "Node memory is filling up. Value = {{ $value | printf `%.2f` }} ({{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint | query | first | value|humanize1024}})"

• 问题：告警通知中转换后的剩余空间数值偏差大（如/run挂载点实际剩余873M，告警显示298.2Mi），且humanize1024函数不符合需求，需要直接计算转换。

解决方案 #

方法一：修正Annotations中的查询逻辑+手动转换 #

数值偏差的核心原因是原query只过滤了mountpoint，未匹配instance和device，导致可能获取到其他实例/设备的指标值。修正后通过完整标签匹配获取当前告警对应的指标，再手动转换为MB：

- alert: HostOutOfDiskSpace_test
  expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes > 0 and ON (instance, device, mountpoint) node_filesystem_readonly == 0 and {instance=~"10.22.22.18.*"}
  for: 1s
  labels:
    severity: high
  annotations:
    summary: "Host {{ $labels.instance }} low disk space \n - Device: {{ $labels.device }} \n - Mountpoint: {{ $labels.mountpoint }}"
    description: "Node disk space is filling up. Used percentage: {{ $value | printf `%.2f` }}% \n Available space: {{ (printf \"node_filesystem_avail_bytes{instance='%s', device='%s', mountpoint='%s'}\" $labels.instance $labels.device $labels.mountpoint | query | first | value) / 1024 / 1024 | printf \"%.2f\" }} MB"

• 关键调整：
  • 给query补充instance和device标签过滤，确保拿到当前告警对应的准确指标值
  • 移除humanize1024，通过/ 1024 / 1024将字节转MB，用printf "%.2f"控制小数位数
  • 修正了summary里的“out of memory”错误（应为disk space）

方法二：在告警表达式中提前计算转换值 #

如果希望$value直接代表剩余空间的MB数，可以修改告警表达式，提前完成单位转换，这样Annotations里直接用$value即可：

- alert: HostOutOfDiskSpace_test
  # 示例：剩余空间低于100MB时告警，可根据需求调整阈值
  expr: (node_filesystem_avail_bytes / 1024 / 1024) < 100 and ON (instance, device, mountpoint) node_filesystem_readonly == 0 and {instance=~"10.22.22.18.*"}
  for: 1s
  labels:
    severity: high
  annotations:
    summary: "Host {{ $labels.instance }} low disk space \n - Device: {{ $labels.device }} \n - Mountpoint: {{ $labels.mountpoint }}"
    description: "Node disk space is filling up. Available space: {{ $value | printf `%.2f` }} MB"

• 优势：无需嵌套query，逻辑更简洁，$value直接是转换后的MB数值
• 可根据实际需求调整告警阈值（比如基于剩余百分比告警，同时保留MB转换逻辑）

内容的提问来源于stack exchange，提问作者Stanislav_iz