Splunk OTEL Collector无法处理File Log Receiver问题求助

问题概述 #

将Azure DevOps Agent日志发送至集群接收器时，出现三类核心错误：

• 正则解析器（parser-azdo）匹配失败
• 级别解析器（azdo-severity）找不到attributes.severity字段
• 字段移动操作（finalize）找不到attributes.message字段

错误日志 #

2025-12-15T17:37:35.848Z error helper/transformer.go:154 Failed to process entry {"resource": {"service.instance.id": "5778d025-1300-4d5e-bd75-bf3645a20f8e", "service.name": "otelcol", "service.version": "v0.140.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "parser-azdo", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send", "entry.timestamp": "0001-01-01T00:00:00.000Z", "log.file.name": "Agent_20251215-173735-utc.log", "log.file.path": "/azp//Agent\_.log"}

2025-12-15T17:37:35.848Z error helper/transformer.go:154 Failed to process entry {"resource": {"service.instance.id": "5778d025-1300-4d5", "service.name": "otelcol", "service.version": "v0.140.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "azdo-severity", "operator_type": "severity_parser", "error": "log entry does not have the expected parse_from field: {"parse_from":"attributes.severity"}", "action": "send", "entry.timestamp": "0001-01-01T00:00:00.000Z", "log.file.name": "Agent_2025121.log", "log.file.path": "/azp/Agent_20121.log"}

2025-12-15T17:37:35.848Z error helper/transformer.go:154 Failed to process entry {"resource": {"service.instance.id": "5778d025-1300-4d5e-bd75", "service.name": "otelcol", "service.version": "v0.140.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "finalize", "operator_type": "move", "error": "move: field does not exist: attributes.message", "action": "send", "entry.timestamp": "0001-01-01T00:00:00.000Z", "log.file.name": "Agent_20251215-173735-utc.log", "log.file.path": "/azp//Agent_20251.log"}

原始日志格式 #

[2025-12-15 11:58:13Z INFO AgentCapabilitiesProvider] Adding 'Agent.OS': 'Linux'
[2025-12-15 11:58:13Z INFO AgentCapabilitiesProvider] Adding 'Agent.OSArchitecture': 'X64'
[2025-12-15 11:58:13Z INFO AgentCapabilitiesProvider] Adding 'InteractiveSession': 'True'
[2025-12-15 11:58:13Z VERB VisualStudioServices] Created OAuth issued token provider instance 65942258 (Bearer)
[2025-12-15 11:58:13Z WARN VisualStudioServices] Authentication failed with status code 401.
Cache-Control: no-cache
Pragma: no-cache
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
WWW-Authenticate: Bearer authorization_uri=https://logi/, Basic realm="https:///",
TFS-Federated
X-TFS-ProcessId: 4983132d-1814-4d88-9060-cf367f589768
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: 3485d06f-

错误原因及修复方案 #

1. 正则匹配失败问题 #

原错误：正则表达式存在语法错误，分组命名缺失、转义符滥用，无法匹配日志格式。
修复后正则：

^\[(?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}Z)\s+(?P<severity>INFO|WARN|VERB|ERROR)\s+(?P<component>[^\]]+)\]\s+(?P<message>[\s\S]*)$

• 为每个捕获组添加明确命名（timestamp/severity/component/message）
• 移除不必要的转义符，修正字符组匹配逻辑
• 确保覆盖完整的标准日志行内容

2. 缺失attributes.severity/attributes.message字段问题 #

原错误：正则解析器未将捕获的字段映射到attributes中，导致后续步骤找不到对应字段。
修复配置：在regex_parser中添加attributes映射，明确指定捕获字段的写入位置：

- type: regex_parser
  id: parser-azdo
  regex: '^\[(?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}Z)\s+(?P<severity>INFO|WARN|VERB|ERROR)\s+(?P<component>[^\]]+)\]\s+(?P<message>[\s\S]*)$'
  timestamp:
    parse_from: attributes.timestamp
    layout_type: gotime
    layout: '2006-01-02 15:04:05Z'
  attributes:
    severity:
      from: severity
    component:
      from: component
    message:
      from: message
  on_error: send
  output: azdo-severity

3. 处理非标准日志行 #

日志中存在Cache-Control这类非[timestamp level component]格式的行，会触发解析错误。添加过滤操作只处理标准格式行：

- type: filter
  id: filter-standard-lines
  condition: 'body startsWith "["'
  output: parser-azdo
  on_error: send

完整修复后的配置 #

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-sidecar-config
  namespace: xxxxxxxxxx
data:
  config.yaml: |
    receivers:
      filelog:
        include:
          - /azp/**.log
        start_at: beginning
        poll_interval: 200ms
        retry_on_failure:
          enabled: true
        include_file_path: true
        include_file_name: true
        operators:
          - type: recombine
            id: azdo-recombine
            source_identifier: attributes["log.file.path"]
            combine_field: body
            combine_with: "\n"
            is_first_entry: 'body startsWith "["'
            max_log_size: 1048576
            output: filter-standard-lines
          - type: filter
            id: filter-standard-lines
            condition: 'body startsWith "["'
            output: parser-azdo
            on_error: send
          - type: regex_parser
            id: parser-azdo
            regex: '^\[(?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}Z)\s+(?P<severity>INFO|WARN|VERB|ERROR)\s+(?P<component>[^\]]+)\]\s+(?P<message>[\s\S]*)$'
            timestamp:
              parse_from: attributes.timestamp
              layout_type: gotime
              layout: '2006-01-02 15:04:05Z'
            attributes:
              severity:
                from: severity
              component:
                from: component
              message:
                from: message
            on_error: send
            output: azdo-severity
          - type: severity_parser
            id: azdo-severity
            parse_from: attributes.severity
            mapping:
              trace: VERB
              info: INFO
              warn: WARN
              error: ERROR
            output: finalize
          - type: move
            id: finalize
            from: attributes.message
            to: body
    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 400
        spike_limit_mib: 100
      k8sattributes:
        extract:
          metadata:
            - k8s.pod.name
            - k8s.namespace.name
        resource:
          attributes:
            - key: service.name
              from_attribute: k8s.pod.name
              action: upsert
      batch: {}
    exporters:
      otlp:
        endpoint: otel-cluster-receiver.splunk.svc.cluster.local:4317
        tls:
          insecure: true
      debug:
        verbosity: detailed
    service:
      telemetry:
        logs:
          level: debug
      pipelines:
        logs:
          receivers: [filelog]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlp]

配置调整说明：

• 修正了原配置中混乱的缩进问题
• 添加过滤操作隔离非标准日志行
• 修复级别解析器的映射值（移除多余的[]）
• 调整处理器结构，将k8sattributes加入日志处理流水线

验证步骤 #

1. 应用修改后的ConfigMap：kubectl apply -f <configmap-file>.yaml
2. 重启OTEL sidecar Pod：kubectl rollout restart deployment <deployment-name> -n <namespace>
3. 查看OTEL日志确认错误消失：kubectl logs <pod-name> -n <namespace>
4. 检查集群接收器是否收到正确格式化的日志

内容的提问来源于stack exchange，提问作者ramesh reddy