寻求可读取File Exporter zstd压缩指标文件的OTel Receiver

在远程主机使用OpenTelemetry Collector的hostmetrics采集指标，通过File Exporter将指标以zstd压缩的proto格式写入文件，传输到本地导入可观测栈时遇到以下问题：

• 使用otlpjsonfile Receiver读取压缩文件失败，但未压缩的proto文件可正常读取
• 标准zstd解压工具无法处理该压缩文件

询问是否存在支持读取此类压缩文件的Receiver。

指标写入配置 #

receivers:
  hostmetrics:
    collection_interval: 1s
    scrapers:
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
  hostmetrics/system:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.logical.count:
            enabled: true
          system.cpu.physical.count:
            enabled: true
          system.cpu.utilization:
            enabled: true
      load:
      memory:
        metrics:
          system.memory.limit:
            enabled: true
          system.memory.utilization:
            enabled: true
          system.linux.memory.available:
            enabled: true
      network:
      process:
        metrics:
          process.cpu.utilization:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.disk.operations:
            enabled: true
      processes:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1000
    spike_limit_mib: 200
  resource:
    attributes:
      - key: host.id
        value: localhost
        action: insert

exporters:
  file/rotation_with_custom_settings:
    path: /scratch/tmplog/metrics.zstd
    rotation:
      max_megabytes: 250
      max_days: 30
      max_backups: 3
      localtime: false
    format: proto
    compression: zstd
  debug:

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    metrics:
      receivers: [hostmetrics, hostmetrics/system]
      processors: [batch, resource, memory_limiter]
      exporters: [file/rotation_with_custom_settings, debug]

指标读取配置 #

receivers:
  otlpjsonfile:
    include:
      - "/tmp/log/*.zstd"

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1000
    spike_limit_mib: 200

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    external_labels:
      instance: nexis_demo_ft
      node: nexis_demo_ft
      job: nexis_demo_ft
  debug:

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    metrics:
      receivers: [otlpjsonfile]
      processors: [batch, memory_limiter]
      exporters: [prometheusremotewrite, debug]

读取日志 #

2024-04-19T18:55:11.260Z    info    fileconsumer/file.go:228    Started watching file   {"kind": "receiver", "name": "otlpjsonfile", "data_type": "metrics", "component": "fileconsumer", "path": "/tmp/log/metrics-2024-04-19T06-33-54.214.zstd"}

解决方案建议 #

1. 使用otlpreceiver配合批次解压

OpenTelemetry Collector的File Exporter启用zstd压缩时，是对每个OTLP批次单独压缩，而非整个文件用zstd打包，因此标准zstd工具无法直接解压。需要编写脚本按批次解压文件，再用otlpreceiver读取解压后的proto文件：

receivers:
  otlp:
    protocols:
      file:
        path: "/tmp/log/uncompressed_metrics/*.proto"

2. 使用filelog Receiver配合解码处理器

如果使用较新版本的OpenTelemetry Collector，可以通过filelog Receiver结合解压和解析处理器直接读取压缩文件：

receivers:
  filelog:
    include:
      - "/tmp/log/*.zstd"
    operators:
      - type: zstd_decompressor
      - type: otlp_proto_parser
        parse_from: body

该方案需确保Collector版本支持zstd_decompressor和otlp_proto_parser操作符。

3. 调整导出配置兼容现有Receiver

如果上述方案不可行，可修改File Exporter配置：

• 移除compression: zstd，使用未压缩的proto格式，直接用otlpreceiver读取
• 改用format: json配合compression: zstd，传输到本地后用标准zstd解压，再用otlpjsonfile Receiver读取

内容的提问来源于stack exchange，提问作者JoeH