如何通过OpenTelemetry Collector与Loki在Grafana中添加K8s日志标签

问题根源 #

1. Loki OTLP映射规则：Loki接收OTLP日志时，默认仅将少量标准属性（如service.name）转为可搜索标签，自定义的pod.name、container.name需要显式配置映射。
2. 处理器执行顺序错误：原配置中处理器顺序导致属性被覆盖或未正确传递，比如k8sattributes应在属性提取后执行，且memory_limiter必须放在最前面。
3. 冗余配置冲突：重复的属性移动/设置操作导致属性未正确生效。

修正后的Helm Values配置 #

mode: daemonset

image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: 0.119.0
  pullPolicy: IfNotPresent

imagePullSecrets: 
  - name: "github-registry-credentials"

extraVolumes:
  - name: varlog
    hostPath:
      path: /var/log

extraVolumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true

config:
  receivers:
    filelog:
      poll_interval: 10s
      include:
        - /var/log/pods/*/*/*.log
      start_at: beginning
      include_file_path: true
      include_file_name: true
      operators:
        - type: regex_parser
          id: extract_metadata
          regex: "/var/log/pods/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_[^/]+/(?P<container_name>.+).log"
          parse_from: attributes["log.file.path"]
          on_error: drop  
        # 直接将提取的属性转为resource属性，避免后续重复操作
        - type: move
          from: attributes["pod_name"]
          to: resource["pod.name"]
        - type: move
          from: attributes["container_name"]
          to: resource["container.name"]
        - type: move
          from: attributes["namespace"]
          to: resource["k8s.namespace.name"]
        - type: add
          field: resource["service.name"]
          value: 'EXPR(resource["k8s.namespace.name"])'

  processors:
    # 内存限制器必须放在所有处理器最前面
    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 15
    # 补充K8s元数据（需确保Collector有ServiceAccount权限）
    k8sattributes:
      auth_type: serviceAccount
      passthrough: false
      extract:
        metadata:
          - k8s.pod.name
          - k8s.container.name
          - k8s.namespace.name
    # 添加自定义标签
    attributes/add_labels:
      actions:
        - action: insert
          key: cluster
          value: cluster-name
    # 过滤无效服务名
    filter/service_name:
      logs:
        exclude:
          match_type: strict
          resource_attributes:
            - key: service.name
              value: "unknown_service"
    # 批量处理优化性能
    batch:
      send_batch_size: 1000
      timeout: 120s

  exporters:
    otlphttp/loki:
      logs_endpoint: https://loki-gateway.domain.com/otlp/v1/logs
      tls:
        insecure: true
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 60s
        max_elapsed_time: 5m
      sending_queue:
        enabled: true
        num_consumers: 10
        queue_size: 5000
      # 关键配置：指定要转为Loki标签的resource属性
      attributes:
        attributes:
          - key: service.name
            action: insert
            value: "{{ resource.attributes['service.name'] }}"
          - key: pod_name
            action: insert
            value: "{{ resource.attributes['k8s.pod.name'] }}"
          - key: container_name
            action: insert
            value: "{{ resource.attributes['k8s.container.name'] }}"
          - key: namespace
            action: insert
            value: "{{ resource.attributes['k8s.namespace.name'] }}"
          - key: cluster
            action: insert
            value: "{{ attributes['cluster'] }}"
        # 将上述属性转为Loki的标签
        loki:
          labels:
            - service.name
            - pod_name
            - container_name
            - namespace
            - cluster

  service:
    extensions:
      - health_check
    pipelines:
      logs:
        receivers: [filelog]
        # 修正处理器顺序：memory_limiter -> k8sattributes -> 属性处理 -> batch
        processors: [memory_limiter, k8sattributes, attributes/add_labels, filter/service_name, batch]
        exporters: [otlphttp/loki]

resources:
  limits:
    cpu: 1
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

关键修改说明 #

• 处理器顺序调整：memory_limiter前置防止OOM，k8sattributes用于补充完整的K8s元数据，避免正则提取的不完整。
• Loki标签映射：在otlphttp/loki exporter中显式配置attributes.loki.labels，指定哪些属性要作为Loki的可搜索标签。
• 简化属性操作：移除重复的transform和resource处理器，直接在filelog接收器中完成属性到resource的移动，减少冲突。

验证步骤 #

1. 重新部署OTel Collector：helm upgrade --install otel-collector open-telemetry/opentelemetry-collector -f values.yaml -n <your-namespace>
2. 查看Collector日志确认无错误：kubectl logs -n <your-namespace> -l app.kubernetes.io/name=opentelemetry-collector -c otel-collector
3. 在Grafana Explore中选择Loki数据源，查看标签列表，确认service.name、pod_name、container_name等标签已显示。

内容的提问来源于stack exchange，提问作者Apkahym