Kubernetes日志在Grafana拆分多行问题（Alloy配置排查）

在Kubernetes环境搭建的Alloy => Loki => Prometheus/Grafana监控栈中，Kubernetes单条错误日志在Grafana Logs中被拆分为多行，严重影响日志搜索效率，需要让每条Kubernetes日志保持为单行。以下是当前的Alloy配置：

loki.write "local" {
  endpoint {
    url       = "http://loki-gateway.monitoring.svc/loki/api/v1/push"
    tenant_id = "Bluey-DEV"
  }
  external_labels = {}
}

logging {
  level  = "info"
  format = "logfmt"
}

loki.process "file_multiline" {
    forward_to = [loki.relabel.k8s_relabel_service_name.receiver]
    stage.multiline {
        firstline = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
        max_wait_time = "10s"
    }
}

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod_name"
  }
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container_name"
  }
}

discovery.kubernetes "services" {
    role = "service"
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.kubernetes_api_multiline.receiver]
}

loki.source.kubernetes "services" {
  targets    = discovery.kubernetes.services.targets
  forward_to = [loki.write.local.receiver]
}

loki.source.kubernetes_events "events" {
  forward_to = [loki.write.local.receiver]
}

local.file_match "k8s_logs" {
  path_targets = [
    {__path__ = "/var/log/kubernetes/*/*.log"},
  ]
}

loki.source.file "kubernetes" {
  targets    = local.file_match.k8s_logs.targets
  forward_to = [loki.process.file_multiline.receiver]
  tail_from_end = true
}

loki.relabel "k8s_relabel_service_name" {
  forward_to = [loki.relabel.k8s_relabels.receiver]
  rule {
    action        = "replace"
    source_labels = ["filename"]
    target_label = "component"
    regex         = `.*\/(.*?)\.log`
    replacement = "$1"
  }
}

loki.relabel "k8s_relabels" {
  forward_to = [loki.write.local.receiver]
  rule {
    action        = "replace"
    source_labels = ["filename"]
    target_label = "k8s_interlnal_svc_log"
    regex         = `.*\/(.*?)\.log`
    replacement = "$1"
  }
}
loki.process "kubernetes_api_multiline" {
  forward_to = [loki.write.local.receiver]

  stage.json {
    expressions = {
      log = "log",
    }
  }

  stage.multiline {
    firstline      = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
    max_wait_time  = "10s"
  }

  stage.output {
    source = "log"
  }
}

你的配置存在多行匹配规则不完整、部分日志处理链路兼容不足的问题，以下是具体修正点：

1. 补全多行日志匹配规则 #

当前的stage.multiline只定义了首行匹配规则，但没有明确标记"哪些行属于同一日志的后续行"，导致异常栈、换行日志无法被正确合并。需要添加反向匹配规则，将所有不符合首行格式的行合并到上一行：

stage.multiline {
  # 匹配标准时间戳开头的行作为日志首行
  firstline = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
  # 反向匹配：所有不满足首行规则的行，都合并到上一条日志
  regex = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
  negate = true
  max_wait_time = "10s"
}

2. 优化JSON日志解析兼容性 #

Kubernetes容器日志可能存在非JSON格式的情况，当前配置如果遇到非JSON日志会直接丢弃，需要添加兼容配置保证日志不丢失：

stage.json {
  expressions = {
    log = "log",
  }
  # 遇到格式错误的日志不丢弃，直接进入下一个处理阶段
  drop_malformed = false
}

3. 修正后的关键配置片段 #

修正后的loki.process "file_multiline" #

loki.process "file_multiline" {
    forward_to = [loki.relabel.k8s_relabel_service_name.receiver]
    stage.multiline {
        firstline = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
        regex = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
        negate = true
        max_wait_time = "10s"
    }
}

修正后的loki.process "kubernetes_api_multiline" #

loki.process "kubernetes_api_multiline" {
    forward_to = [loki.write.local.receiver]

    stage.json {
        expressions = {
            log = "log",
        }
        drop_malformed = false
    }

    stage.multiline {
        firstline = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
        regex = `^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}`
        negate = true
        max_wait_time = "10s"
    }

    stage.output {
        source = "log"
    }
}

内容的提问来源于stack exchange，提问作者Vincent H