OpenSearch临时不可用时，如何避免Otel Collector日志丢失？

场景与需求 #

• 日志生成规则：控制台应用每5分钟生成一个命名格式为testlog-YYYY-MM-DD-HH-mm-ss.log的日志文件，每分钟追加约10条日志
• 部署架构：OpenTelemetry Collector（镜像otel/opentelemetry-collector-contrib:latest）以Sidecar/Deployment模式运行，通过filelogreceiver采集日志，再通过OpenSearch Exporter推送至OpenSearch Pod
• 核心问题：OpenSearch Pod崩溃或重启（约2-3分钟不可用）期间，Collector处理的日志丢失，需实现OpenSearch恢复后，缓存的离线日志与新生成日志一并推送

当前简化配置 #

config:  
  exporters:
    opensearch/log:
      logs_index: ss4o_log-otel-test  
      http:
        endpoint: https://opensearch-cluster-master.opensearch.svc.cluster.local:9200  
        auth:
          authenticator: basicauth/client
        tls:
          insecure: false
          insecure_skip_verify: true      
      sending_queue:
        enabled: true
        num_consumers: 2
        queue_size: 5000 
        storage: file_storage    
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 30s
        max_elapsed_time: 300s
    debug:
      verbosity: detailed
  extensions:
    file_storage:
      directory: /var/lib/otelcol/file_storage  
      timeout: 5s           
      create_directory: true           
      compaction:
        on_start: true    
        directory: /var/lib/otelcol/storage           
        check_interval: 5m
    basicauth/client:
      client_auth:
        username: admin
        password: TadhakDev**789
    health_check:
      endpoint: ${env:MY_POD_IP}:13133
  processors:
    batch:
      timeout: 5s
      send_batch_size: 10000    
    memory_limiter:     
      check_interval: 5s     
      limit_percentage: 80    
      spike_limit_percentage: 25

  receivers:
    filelog:
      include:
        - /var/log/testlog-*.log
      retry_on_failure:
        enabled: true        
      storage: file_storage
      start_at: beginning  
      include_file_name: true
      include_file_path: true     
      operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.@t
          layout: '%Y-%m-%dT%H:%M:%S.%fZ'    
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318

  service:
    telemetry:
      metrics:
        readers:
          - pull:
              exporter:
                prometheus:
                  host: ${env:MY_POD_IP}
                  port: 8888
    extensions:
      - file_storage
      - health_check
      - basicauth/client
    pipelines:
      logs:
        receivers: [filelog]
        processors: [batch]
        exporters: [opensearch/log]

extraVolumes:
  - name: log-volume-new
    persistentVolumeClaim:
      claimName: log-storage-pvc1
  - name: otel-storage
    persistentVolumeClaim:
      claimName: otel-storage-pvc
  - name: cleaner-script
    configMap:
      name: otel-log-cleaner-script
extraVolumeMounts:
  - name: log-volume-new
    mountPath: /var/log   
  - name: otel-storage
    mountPath: /var/lib/otelcol/file_storage
    
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: otel-storage-pvc
  namespace: opensearch
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

优化配置与关键调整 #

针对日志丢失问题，需重点优化重试策略、持久化队列和批量处理三个核心环节，调整后的关键配置如下：

1. 调整OpenSearch Exporter的重试与队列配置 #

将retry_on_failure.max_elapsed_time设为0（无限重试直到成功），确保OpenSearch恢复前不会丢弃日志；同时适当增大队列容量以应对更长时间的离线场景：

exporters:
  opensearch/log:
    # ... 其他配置保持不变
    sending_queue:
      enabled: true
      num_consumers: 2
      queue_size: 10000  # 增大队列容量，适配更长离线时间
      storage: file_storage    
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 0  # 无限重试，直到OpenSearch恢复可用

2. 优化Batch处理器参数 #

过大的批量发送尺寸可能导致OpenSearch恢复初期请求超时，调整为更合理的批次大小：

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000  # 缩小批量尺寸，降低推送超时风险
    send_batch_max_size: 2000  # 新增最大批量限制，避免单批次过大

3. 确保File Storage扩展配置正确 #

当前file_storage的compaction.directory与主存储目录分离，需确保该目录也挂载到持久化存储，避免Collector重启后丢失压缩的缓存数据：

extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage  
    timeout: 5s           
    create_directory: true           
    compaction:
      on_start: true    
      directory: /var/lib/otelcol/file_storage/compaction  # 修改为持久化目录下的子目录
      check_interval: 5m

（当前挂载已覆盖父目录，此修改无需额外调整VolumeMount配置）

4. 验证Filelog Receiver的持久化配置 #

确认filelog接收器已正确绑定file_storage扩展，确保采集进度持久化，避免重复采集或丢失日志位置：

receivers:
  filelog:
    # ... 其他配置保持不变
    storage: file_storage  # 已正确配置，无需修改

关键原理说明 #

• 持久化队列：通过file_storage扩展将待推送日志存储到PVC中，避免Collector重启或内存不足时丢失数据
• 无限重试：max_elapsed_time: 0确保OpenSearch不可用期间，Collector不会终止重试，直到服务恢复
• 合理批量：缩小批量尺寸降低推送失败概率，提升恢复后的推送效率

内容的提问来源于stack exchange，提问作者Dev_NG26