OpenTelemetry FileLog Receiver在Exporter故障时丢失日志求助

问题描述 #

我的场景为：每3分钟生成多个日志文件，每个文件每分钟约包含10条日志条目。

日志文件示例：

testlog-2025-12-04-12-00-00.log
testlog-2025-12-04-12-03-00.log
testlog-2025-12-04-12-06-00.log

这些日志文件由OpenTelemetry Collector通过FileLog Receiver采集、处理后导出至OpenSearch。

问题 #

当OpenSearch正常运行时，一切工作正常。但如果OpenSearch Pod崩溃或无法访问，Exporter无法发送日志。

在此故障场景下，若设置delete_after_read: true，文件会在读取后立即被删除，即便日志并未成功导出。

我需要相反的行为：仅当日志成功导出至OpenSearch后，才删除日志文件。

我已尝试：

• 使用文件存储检查点标记
• 编写Shell脚本检查哪些文件仍在被读取

但仍无法可靠检测日志是否已实际导出。

需求 #

仅在以下条件满足时删除日志文件：

• FileLog Receiver已完成文件读取，且
• Exporter（OpenSearch）已成功发送所有日志

以下情况不应删除日志文件：

• Exporter故障
• OpenSearch Pod崩溃
• Collector无法发送剩余日志

nameOverride: ""
fullnameOverride: ""

mode: "deployment"

namespaceOverride: ""

presets:

    logsCollection:
        enabled: false
        includeCollectorLogs: false

        storeCheckpoints: true

        maxRecombineLogSize: 102400

    hostMetrics:
        enabled: false

    kubernetesAttributes:
        enabled: false

        extractAllPodLabels: false

        extractAllPodAnnotations: false

    kubeletMetrics:
        enabled: false

    kubernetesEvents:
        enabled: false

    clusterMetrics:
        enabled: false

    annotationDiscovery:
        logs:
            enabled: false
        metrics:
            enabled: false

configMap:
    create: true

    existingName: ""

internalTelemetryViaOTLP:

    endpoint: ""
    headers: []

    traces:
        enabled: false

        endpoint: ""

        headers: []
    metrics:
        enabled: false

        endpoint: ""

        headers: []
    logs:
        enabled: false

        endpoint: ""

        headers: []

config:

    exporters:
        opensearch:
            logs_index: ss4o_log-otel-timedloggermvc
            http:
                endpoint: https://opensearch-cluster-master.open.svc.cluster.local:9200
                auth:
                    authenticator: basicauth/client
                tls:
                    insecure_skip_verify: true

            sending_queue:
                enabled: true
                storage: file_storage
                queue_size: 20000
                num_consumers: 2

            retry_on_failure:
                enabled: true
                max_elapsed_time: 0   # retry forever

        debug: {}
    extensions:
        file_storage:
            directory: /var/lib/otelcol/checkpoints
            create_directory: true
        basicauth/client:
            client_auth:
                username: admin
                password: Tadhak**Dev02

        health_check:
            endpoint: ${env:MY_POD_IP}:13133
    processors:
        batch:
            send_batch_size: 50
            timeout: 5s

        memory_limiter:

            check_interval: 5s

            limit_percentage: 80

            spike_limit_percentage: 25

    receivers:
        filelog:
            include:
                - /var/log/tadhak/tadhak-*.log
            start_at: beginning
            include_file_name: true
            include_file_path: true
            delete_after_read: false
            storage: file_storage
            operators:
                - type: json_parser
                  timestamp:
                    parse_from: attributes.@t
                    layout: '%Y-%m-%dT%H:%M:%S.%fZ'

        otlp:
            protocols:
                grpc:
                    endpoint: ${env:MY_POD_IP}:4317
                http:
                    endpoint: ${env:MY_POD_IP}:4318
        # if internalTelemetryViaOTLP.metrics.enabled = true, prometheus receiver will be removed
        prometheus:
            config:
                scrape_configs:
                    - job_name: opentelemetry-collector
                      scrape_interval: 10s
                      static_configs:
                        - targets:
                            - ${env:MY_POD_IP}:8888
        zipkin:
            endpoint: ${env:MY_POD_IP}:9411
    service:
        telemetry:
            metrics:
                readers:
                    - pull:
                        exporter:
                          prometheus:
                            host: ${env:MY_POD_IP}
                            port: 8888
        extensions:
            - health_check
            - basicauth/client
            - file_storage
        pipelines:
            logs:
                receivers: [filelog]
                processors: [memory_limiter, batch]   # 🔥 REQUIRED
                exporters: [opensearch] 

alternateConfig: {}

image:

    repository: "otel/opentelemetry-collector-contrib"
    pullPolicy: IfNotPresent

    tag: "latest"

    digest: ""
imagePullSecrets: []

command:
    name: otelcol-contrib
    extraArgs:
        - "--feature-gates=filelog.allowFileDeletion"

serviceAccount:
    create: true
    annotations: {}
    name: ""
    automountServiceAccountToken: true

clusterRole:
    create: false
    annotations: {}
    name: ""

    rules: []

    clusterRoleBinding:

        annotations: {}

        name: ""

podSecurityContext: {}
securityContext: {}

nodeSelector: {}
tolerations: []
affinity: {}
topologySpreadConstraints: []

priorityClassName: ""

runtimeClassName: ""

extraEnvs: []
extraEnvsFrom: []

extraVolumes:
    - name: log-volume
      persistentVolumeClaim:
          claimName: log-storage-pvc1 # Use Nginx logs PVC

    - name: otel-storage
      persistentVolumeClaim:
          claimName: otel-storage-pvc

    - name: cleanup-script
      configMap:
          name: log-cleanup-script
          defaultMode: 0777

extraVolumeMounts:
    - name: log-volume
      mountPath: /var/log/tadhak  # Match Nginx log directory
      # readOnly: true  # OpenTelemetry should only read logs

    - name: otel-storage
      mountPath: /var/lib/otelcol/checkpoints

extraManifests: []

ports:
    otlp:
        enabled: true
        containerPort: 4317
        servicePort: 4317
        hostPort: 4317
        protocol: TCP
        # nodePort: 30317
        appProtocol: grpc
    otlp-http:
        enabled: true
        containerPort: 4318
        servicePort: 4318
        hostPort: 4318
        protocol: TCP
    jaeger-compact:
        enabled: true
        containerPort: 6831
        servicePort: 6831
        hostPort: 6831
        protocol: UDP
    jaeger-thrift:
        enabled: true
        containerPort: 14268
        servicePort: 14268
        hostPort: 14268
        protocol: TCP
    jaeger-grpc:
        enabled: true
        containerPort: 14250
        servicePort: 14250
        hostPort: 14250
        protocol: TCP
    zipkin:
        enabled: true
        containerPort: 9411
        servicePort: 9411
        hostPort: 9411
        protocol: TCP
    metrics:
        # The metrics port is disabled by default. However you need to enable the port
        # in order to use the ServiceMonitor (serviceMonitor.enabled) or PodMonitor (podMonitor.enabled).
        enabled: false
        containerPort: 8888
        servicePort: 8888
        protocol: TCP

useGOMEMLIMIT: true

resources: {}

enableConfigChecksumAnnotation: true
podAnnotations: {}

podLabels: {}

additionalLabels: {}

hostNetwork: false

hostAliases: []

dnsPolicy: ""

dnsConfig: {}

schedulerName: ""

replicaCount: 1

revisionHistoryLimit: 10

annotations: {}

extraContainers: []

initContainers: []

lifecycleHooks: {}

livenessProbe:

    httpGet:
        port: 13133
        path: /

readinessProbe:

    httpGet:
        port: 13133
        path: /

startupProbe: {}

service:

    type: ClusterIP
    annotations: {}
ingress:
    enabled: false

    additionalIngresses: []

podMonitor:
    enabled: false
    metricsEndpoints:
        - port: metrics

    extraLabels: {}

serviceMonitor:

    enabled: false
    metricsEndpoints:
        - port: metrics

    extraLabels: {}

    relabelings: []
    metricRelabelings: []

podDisruptionBudget:
    enabled: false

autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 10
    behavior: {}
    targetCPUUtilizationPercentage: 80
    additionalMetrics: []

rollout:
    rollingUpdate: {}
    strategy: RollingUpdate

prometheusRule:
    enabled: false
    groups: []
    defaultRules:
        enabled: false

    extraLabels: {}

statefulset:
    volumeClaimTemplates: []
    podManagementPolicy: "Parallel"
    persistentVolumeClaimRetentionPolicy:
        enabled: false
        whenDeleted: Retain
        whenScaled: Retain

networkPolicy:
    enabled: false

    annotations: {}

    allowIngressFrom: []

    extraIngressRules: []

    egressRules: []

shareProcessNamespace: false

日志生成模式 #

我的应用程序行为如下：

• 每3分钟生成一个新日志文件
• 每分钟写入5条日志

示例：

这些日志最初可成功导入OpenSearch。

故障场景 #

OpenSearch Pod崩溃，约7分钟无法访问

在此停机期间，我的应用程序继续写入日志：

OpenSearch恢复后，仅重启后的新日志（如56–60）被发送至OpenSearch，日志21–55从未被导入，即便：

• 文件仍然存在
• 磁盘上存在检查点
• 已启用Exporter重试机制

内容的提问来源于stack exchange，提问作者om makwana