安装Bitnami Elasticsearch Helm Chart后无法获取节点锁求助

背景 #

此前已成功使用 NFS-CSI 驱动为 bitnami/kafka 提供持久化存储，但在为 bitnami/elasticsearch 配置相同存储方案时启动失败。

当前存储类信息 #

$ kubectl get sc
NAME                PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi (default)   nfs.csi.k8s.io    Delete          Immediate           false                  43d

安装步骤 #

$ kubectl create ns elk2

$ helm pull oci://registry-1.docker.io/bitnamicharts/elasticsearch

$ vim elasticsearch/values.yaml

修改后的values.yaml核心配置：

global:
  imageRegistry: ""
  imagePullSecrets: []
  storageClass: "nfs-csi"   
  elasticsearch:
    service:
      name: elasticsearch
      ports:
        restAPI: 9200
  kibanaEnabled: false

执行安装：

$ helm install elasticsearch ./ -f values.yaml -n elk2 

$ helm list -n elk2
NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                   APP VERSION
elasticsearch   elk2        1           2023-07-28 22:26:03.468994841 +0800 CST deployed    elasticsearch-19.10.6   8.9.0    

错误日志 #

$ kubectl logs -f elasticsearch-master-1 -n elk2
.....
.....
[2023-07-28T14:29:08,437][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [vector-tile]
[2023-07-28T14:29:08,437][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-expression]
[2023-07-28T14:29:08,437][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-eql]
[2023-07-28T14:29:10,660][ERROR][o.e.b.Elasticsearch      ] [elasticsearch-master-1] fatal exception while booting Elasticsearch java.lang.IllegalStateException: failed to obtain node locks, tried [/bitnami/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:292)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:490)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.node.Node.<init>(Node.java:334)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72)
Caused by: java.io.IOException: failed to obtain lock on /bitnami/elasticsearch/data
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:237)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:205)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:284)
    ... 5 more
Caused by: java.nio.file.NoSuchFileException: /bitnami/elasticsearch/data/node.lock
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:825)
    at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)
    at org.apache.lucene.core@9.7.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
    at org.apache.lucene.core@9.7.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
    at org.elasticsearch.server@8.9.0/org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:230)
    ... 7 more
    Suppressed: java.nio.file.AccessDeniedException: /bitnami/elasticsearch/data/node.lock
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
        at java.base/java.nio.file.Files.createFile(Files.java:658)
        at org.apache.lucene.core@9.7.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)
        ... 10 more

ERROR: Elasticsearch did not exit normally - check the logs at /opt/bitnami/elasticsearch/logs/elastic.log

ERROR: Elasticsearch exited unexpectedly

已完成排查 #

• 确认 NFS 服务器端目录权限配置正常

解决方案 #

1. 替换 Elasticsearch 文件锁实现 #

NFS 对NativeFSLockFactory的支持存在兼容性问题，改用SimpleFSLockFactory规避：
在values.yaml中添加配置：

elasticsearch:
  config:
    env.lock_factory: "simplefs"

2. 调整 NFS 挂载参数 #

修改 NFS-CSI 存储类的挂载参数，添加文件锁相关配置：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: <你的NFS服务器地址>
  share: <你的NFS共享路径>
  mountOptions: "nolock"  # 若NFS版本为v4可尝试改为"lock"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: false

3. 确认容器内目录权限 #

Bitnami Elasticsearch 默认以1001用户运行，在 NFS 服务器上调整目录权限：

chown -R 1001:1001 /path/to/nfs/share/elasticsearch-pvc
chmod -R 755 /path/to/nfs/share/elasticsearch-pvc

临时排查可尝试以 root 用户启动容器（不建议生产环境使用）：

elasticsearch:
  master:
    securityContext:
      runAsUser: 0
      runAsGroup: 0

4. 确保节点使用独立 PVC #

验证每个 Elasticsearch 节点都分配了独立的 PVC，检查values.yaml配置：

elasticsearch:
  master:
    persistence:
      enabled: true
      storageClass: "nfs-csi"
  data:
    persistence:
      enabled: true
      storageClass: "nfs-csi"

内容的提问来源于stack exchange，提问作者Wei Yu