Prometheus告警规则动态变量配置与内存限制展示咨询

原始告警规则 #

- alert: high_memory_usage_resource_limits
  expr: >
    (
      sum by (container, pod, namespace) (container_memory_usage_bytes)
    ) 
    / 
    (
      sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory"})
    ) * 100 > 90
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "High memory usage compared to resource limits on {{$labels.pod}}"
    description: "Container {{$labels.container}} in pod {{$labels.pod}} is consuming more memory than its resource limit ({{$value}}% of the limit)"

问题1：能否将阈值配置为变量并在告警消息中复用？ #

可以实现，推荐两种实用方式：

方式1：自定义标签复用阈值 #

无需修改Prometheus全局配置，直接在告警规则的labels中定义阈值标签，后续在annotations中引用即可：

- alert: high_memory_usage_resource_limits
  expr: >
    (
      sum by (container, pod, namespace) (container_memory_usage_bytes)
    ) 
    / 
    (
      sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory"})
    ) * 100 > 90
  for: 1m
  labels:
    severity: warning
    memory_threshold: "90" # 定义阈值标签
  annotations:
    summary: "High memory usage compared to resource limits on {{$labels.pod}}"
    description: "Container {{$labels.container}} in pod {{$labels.pod}} is consuming {{$value}}% of its memory limit, exceeding the threshold of {{$labels.memory_threshold}}%"

方式2：全局变量复用阈值 #

如果需要在多个告警规则中共享同一阈值，可在Prometheus全局配置中定义变量：

1. 在prometheus.yml添加全局变量：

global:
  external_labels:
    memory_threshold: 90

2. 修改告警规则引用变量：

- alert: high_memory_usage_resource_limits
  expr: >
    (
      sum by (container, pod, namespace) (container_memory_usage_bytes)
    ) 
    / 
    (
      sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory"})
    ) * 100 > $memory_threshold
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "High memory usage compared to resource limits on {{$labels.pod}}"
    description: "Container {{$labels.container}} in pod {{$labels.pod}} is consuming {{$value}}% of its memory limit, exceeding the threshold of {{$external_labels.memory_threshold}}%"

问题2：能否在告警消息中展示内存限制值？ #

可以通过Prometheus模板函数humanize将原始字节数转换为可读格式（如1Gi、512Mi），直接在annotations中查询并展示内存限制：

- alert: high_memory_usage_resource_limits
  expr: >
    (
      sum by (container, pod, namespace) (container_memory_usage_bytes)
    ) 
    / 
    (
      sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory"})
    ) * 100 > 90
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "High memory usage compared to resource limits on {{$labels.pod}}"
    description: |
      Container {{$labels.container}} in pod {{$labels.pod}} is consuming {{$value}}% of its memory limit.
      Memory limit: {{ humanize (sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory"})) }}

humanize函数会自动将字节单位转换为最直观的可读格式，比如将1073741824字节转换为1Gi。

内容的提问来源于stack exchange，提问作者Daniel