如何将Azure日志片段传入Grafana告警规则消息？

我配置了Grafana告警规则监控Azure中的Jira服务日志，告警能正常触发并发送到指定渠道，但始终无法将Azure中的异常日志内容提取并注入到告警消息里。之前尝试在查询的summarize阶段加入message字段，结果导致规则无法在Grafana中运行，但这个写法能体现我的需求：

- alert: errorUnexpected
  for: 1m
  subscription: "/subscriptions/SUBSCRIPTION"
  query_type: "Azure Log Analytics"
  azure_log_analytics:
    query: |           
      ContainerLogV2
      | where $__timeFilter(TimeGenerated)
      | where ContainerId in (
          KubePodInventory
          | where TenantId   == "ID"
          | where ServiceName == "jira"
          | project ContainerId = ContainerID
      )
      | where LogLevel in ("error", "unknown")
      | where tostring(LogMessage) matches regex @"(?i)Unexpected error"
      | extend message = replace_string(substring(tostring(LogMessage), 0, 300), "\n", " ")
      | summarize EventsPerMinute = count() by bin(TimeGenerated, 1m), message
      | order by TimeGenerated asc    
    resources:
      - /subscriptions/SUBSCRIPTION
  threshold: 0
  labels:
    disabled_resolved: true
    severity: critical
    category: logs
    message: "{{ if $labels.message }}{{ $labels.message }}{{ end }}"
  annotations:
    description: "Unexpected error in jira service: {{ $labels.message }}"
    summary: "Unexpected error in jira service"

问题根源 #

Grafana的Azure Log Analytics告警要求查询返回单维度的时间序列指标（用于和阈值对比），之前的写法在summarize里按message分组，会让每条不同的日志内容生成一条独立的时间序列，破坏了告警规则的执行逻辑，导致规则无法运行。

修正后的配置方案 #

调整查询逻辑，把同一分钟内的异常日志内容聚合为单个字段，同时保证summarize只按时间维度分组，保留单指标序列：

- alert: errorUnexpected
    for: 1m
    subscription: "/subscriptions/SUBSCRIPTION"
    query_type: "Azure Log Analytics"
    azure_log_analytics:
      query: |           
        ContainerLogV2
        | where $__timeFilter(TimeGenerated)
        | where ContainerId in (
            KubePodInventory
            | where TenantId   == "ID"
            | where ServiceName == "jira"
            | project ContainerId = ContainerID
        )
        | where LogLevel in ("error", "unknown")
        | where tostring(LogMessage) matches regex @"(?i)Unexpected error"
        | extend trimmed_msg = replace_string(substring(tostring(LogMessage), 0, 300), "\n", " ")
        | summarize 
            EventsPerMinute = count(),
            error_messages = make_list(trimmed_msg) 
          by bin(TimeGenerated, 1m)
        | extend error_messages_str = strcat_array(error_messages, " | ")
        | order by TimeGenerated asc    
      resources:
        - /subscriptions/SUBSCRIPTION
    threshold: 0
    labels:
      disabled_resolved: true
      severity: critical
      category: logs
    annotations:
      description: "Jira服务出现意外错误，共{{ $value }}条异常日志：{{ $values.error_messages_str }}"
      summary: "Jira服务触发意外错误告警"

关键调整点 #

1. 聚合日志内容：用make_list(trimmed_msg)把同一分钟的所有异常日志收集成数组，再通过strcat_array转为字符串，避免按日志内容分组产生多序列问题。
2. 模板变量引用：在annotations里用{{ $values.error_messages_str }}直接调用聚合后的日志内容，Grafana会自动提取查询结果中的对应字段。
3. 保留单指标序列：summarize仅按TimeGenerated分组，确保返回的是用于阈值判断的单指标时间序列（EventsPerMinute），符合Grafana告警规则的要求。

内容的提问来源于stack exchange，提问作者Yonoss