Promtail-Loki-Grafana中logfmt日志时间未生效问题求助

问题说明 #

日志采用logfmt格式，包含time字段，但在Grafana中查看时，显示的是日志上传到Loki的时间，而非日志自身记录的时间。

Docker Compose配置 #

version: "3"

networks:
  tracing:

services:
  grafana:
    container_name: grafana-2
    image: grafana/grafana:10.4.1
    volumes:
      - './run/resources/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yaml'
      - './dockerVolumes/grafana_data:/var/lib/grafana'
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - TZ=UTC
    ports:
      - "3000:3000"
    networks:
      - tracing

  loki:
    container_name: loki
    image: grafana/loki:2.9.6
    volumes:
      - './dockerVolumes/loki_data:/loki'
    environment:
      - TZ=UTC
    ports:
      - "3100:3100"
    networks:
      - tracing

  promtail:
    container_name: promtail
    image: grafana/promtail:2.9.6
    volumes:
      - './run/resources/promtail-config.yml:/etc/promtail/config.yml'
      - './run/logs:/var/log'
    command: -config.file=/etc/promtail/config.yml
    environment:
      - TZ=UTC
    networks:
      - tracing

Promtail配置 #

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: od
    static_configs:
      - targets:
          - localhost
        labels:
          job: od_logs
          __path__: /var/log/od/*.log
    relabel_configs:
      - source_labels: ['__path__']
        regex: '/var/log/od/(.*)\.log'
        action: replace
        target_label: logfilename
        replacement: '${1}'
    pipeline_stages:
      - logfmt:
          mapping:
            timestamp: time
      - timestamp:
          source: time
          format: RFC3339

Loki配置 #

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  max_transfer_retries: 0
  rate_limit_bytes: 8388608
  chunk_idle_period: 1h
  chunk_retain_period: 30s
  lifecycler:
    address: localhost
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  client:
    backoff_config:
      max_period: 5s


schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
    index_queries_cache_config:
      cache:
        max_size: 50000
        item_size_bytes: 5000
        max_age: 1h
    index_queries_cache:
      enabled: true

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem

limits_config:
  enforce_metric_name: false
  reject_old_samples: false
  reject_old_samples_max_age: 999999h 
  unordered_writes: true

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

示例日志 #

time=2024-04-04T18:15:55.434 level=trace thread=3214 origin=libOD msg="9000"
time=2012-11-01T22:08:41+00:00 level=trace thread=3214 origin=libOD msg="9000"
time=2024-04-04T18:15:55.434 level=trace thread=3214 origin=libOD msg="event processed successfully"

异常截图 #

问题原因与解决方法 #

原因 #

示例日志存在两种时间格式：

1. 无时区格式：2024-04-04T18:15:55.434
2. 带时区格式：2012-11-01T22:08:41+00:00

当前Promtail配置仅指定format: RFC3339，该格式要求必须带时区信息，导致无时区的日志解析失败。解析失败时，Promtail会自动 fallback 为日志的采集上传时间，从而出现时间显示异常。

解决方法 #

修改Promtail的pipeline_stages，支持多种时间格式，并指定默认时区：

pipeline_stages:
  - logfmt:  # 移除不必要的timestamp映射，logfmt会自动解析time字段
  - timestamp:
      source: time
      multiple_formats:
        - RFC3339
        - "2006-01-02T15:04:05.000"  # 匹配无时区的时间格式
      loc: UTC  # 指定默认时区为UTC

修改后，Promtail能正确解析两种格式的时间字段，日志将使用自身记录的时间戳在Grafana中显示。

内容的提问来源于stack exchange，提问作者Hector .Barragan