如何在Promtail配置中添加自定义标签？sync_pair与JobId提取问题

问题概述 #

需从目标日志中提取sync_pair和JobId作为Promtail日志标签，但已配置的pipeline stages未生效，标签未出现在日志元数据中。

日志样例 #

2022-12-21T11:48:00,001 [schedulerFactor_Worker-4, , ] INFO  [,,] [userAgent=] [system=,component=,object=] [,] [] c.s.f.s.scheduler.SchedulerTask - sync process started on  2022-12-21T06:48:00.000780  for sync pair :17743b1b-a067-4478-a6d8-7b1cff04175a for JobId :dc8dc0dd-fdb9-4873-af55-1c70ba2047a5

现有配置的核心问题 #

1. 正则表达式无效：当前正则仅匹配sync_pair和job_id字符串本身，未提取日志中UUID格式的实际值
2. pipeline stages归属错误：配置中将stages放在dockerlogs job下，但目标日志属于system job（job=varlogs），导致规则未被应用
3. 标签名不匹配：日志中字段为JobId，但配置中用job_id，命名不一致导致无法映射

修正后的完整配置 #

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  http_listen_address: 0.0.0.0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  # 将pipeline stages移至system job下，针对varlogs日志处理
  pipeline_stages:
  - regex:
      # 匹配日志中sync pair和JobId后的UUID值
      expression: 'for sync pair :(P<sync_pair>[0-9a-fA-F-]+) for JobId :(P<JobId>[0-9a-fA-F-]+)'
  - labels:
      sync_pair:
      JobId:
- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containers
      __path__: /var/lib/docker/containers/*/*log
- job_name: dockerlogs
  file_sd_configs:
  - files:
    - /etc/promtail/promtail-targets.yml
  relabel_configs:
  - source_labels: [job]
    target_label: job
  - source_labels: [__address__]
    target_label: container_id
  - source_labels: [container_id]
    target_label: __path__
    replacement: /var/lib/docker/containers/*/*log

生效验证 #

1. 重启Promtail服务，加载新配置
2. 查看Loki日志界面或通过LogQL查询（如{job="varlogs"} | sync_pair）确认标签是否存在
3. 可使用Promtail dry-run模式测试pipeline效果：

   promtail -config.file=promtail-config.yml -dry-run
   

内容的提问来源于stack exchange，提问作者Sneha Singh