AKS集群Filebeat Pod因无法获取锁文件导致CrashLoopBackOff故障

我通过helm仓库elastic https://helm.elastic.co部署ELK集群，已成功完成Elasticsearch部署：

helm install elasticsearch elastic/elasticsearch --namespace=logging

随后执行以下命令部署Filebeat：

helm install filebeat elastic/filebeat --namespace=logging --values filebeat-values.yaml

filebeat-values.yaml配置内容如下：

daemonset:
  extraEnvs:
    - name: "ELASTICSEARCH_USERNAME"
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: username
    - name: "ELASTICSEARCH_PASSWORD"
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password

filebeatConfig:
  filebeat.yml: |
    logging.metrics.enabled: false
    filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/agri-check*.log
        json:
          keys_under_root: true
          overwrite_keys: true
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"
      - type: container
        paths:
          - /var/log/containers/*.log
        exclude_files: ['.*/agri-check.*$']
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"

    output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: "https://elasticsearch-master:9200"
      username: '${ELASTICSEARCH_USERNAME}'
      password: '${ELASTICSEARCH_PASSWORD}'
      protocol: https
      ssl.verification_mode: none

当前有一个Filebeat Pod处于崩溃状态：

filebeat-filebeat-x74hz     0/1     CrashLoopBackOff   6 (3m7s ago)    9m2s

错误日志显示：

{"log.level":"info","@timestamp":"2023-07-13T08:55:56.776Z","log.origin":{"file.name":"instance/beat.go","file.line":392},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-07-13T08:55:56.776Z","log.origin":{"file.name":"instance/beat.go","file.line":1057},"message":"Exiting: cannot obtain lockfile: connot start, data directory belongs to process with pid 8","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: cannot obtain lockfile: connot start, data directory belongs to process with pid 8

仅该Pod异常，其他节点上的Pod运行正常，且QA集群使用相同配置无此问题，已确认与Elasticsearch的连接正常，无法进入停止的Pod删除锁文件，需排查解决方法。

排查与解决方法 #

• 重建异常Pod：DaemonSet会自动在对应节点重新创建Pod，重建过程会清理旧容器的残留资源，大概率解决锁文件占用问题。执行命令：

  kubectl delete pod filebeat-filebeat-x74hz -n logging
  

• 清理节点残留进程：登录出问题的K8s节点，查找PID为8的进程是否是残留的Filebeat进程：

  ps -ef | grep 8 | grep filebeat
  

  若存在该进程，直接强制终止：

  kill -9 8
  

  之后再重建Pod即可。
• 手动删除节点上的锁文件：Filebeat锁文件默认存放在数据目录（容器内通常为/usr/share/filebeat/data，若用HostPath挂载则对应节点上的实际路径），登录节点后找到该目录，删除.lock后缀的锁文件。
• 禁用Filebeat锁文件：在filebeat-values.yaml的filebeatConfig中添加禁用锁文件的配置，然后执行helm升级：

  filebeatConfig:
    filebeat.yml: |
      # 保留原有配置，新增以下内容
      filebeat.registry.lockfile: ""
  

  升级命令：

  helm upgrade filebeat elastic/filebeat -n logging --values filebeat-values.yaml
  

• 检查节点存储权限与状态：确认该节点上Filebeat数据卷的权限是否与容器内用户匹配，或存储介质是否存在读写故障。执行命令检查目录权限：

  ls -ld /path/to/filebeat/data/dir
  

内容的提问来源于stack exchange，提问作者mbieren