GKE集群单个Elasticsearch pod遇CrashLoopBackOff问题求助

问题描述 #

在GKE部署3节点Elasticsearch集群，其中一个Pod处于CrashLoopBackOff状态无法恢复，日志抛出java.lang.IllegalArgumentException，提示unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings，另外两个Pod运行正常。需排查：

1. 该问题是否由YAML中install-plugins配置导致？
2. 为何仅单个Pod出现异常？
3. 如何解决该问题？

异常日志 #

{
    "type": "server", "timestamp": "2022-08-29T19:52:29,743Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "dev", "node.name": "dev-es-data-hot-1", "message": "uncaught exception in thread [main]", 
    "stacktrace": [
        "org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
        "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "Caused by: java.lang.IllegalArgumentException: unknown secure setting [dev-es-snapshot-backup-feeb83405c27.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
        "at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "uncaught exception in thread [main]",
        "at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:137) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.node.Node.<init>(Node.java:500) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]",
        "... 6 more"
    ]
}

YAML配置 #

- name: data-hot-ingest
  count: 3
  config:
    node.roles: ["data_hot", "ingest", "data_content"]
    node.attr.data: hot
    node.store.allow_mmap: false
    xpack.security.authc:
      anonymous:
        username: anon
        roles: monitoring_user
  podTemplate:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In
                values:
                - hot
      initContainers:
      - name: install-plugins
        command:
        - sh
        - -c
        - |
          bin/elasticsearch-plugin install --batch repository-gcs
      - name: set-virtual-mem
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
      containers:
        - name: elasticsearch
          resources:
            requests:
              memory: "64Gi"
              cpu: "30000m"
            limits:
              memory: "65Gi"
              cpu: "30000m"
          env:
            - name: ES_JAVA_OPTS
              value: -Xms32g -Xmx32g
          readinessProbe:
            httpGet:
              scheme: HTTPS
              port: 8080
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 350Gi
      storageClassName: gold

补充说明 #

配置了关联集群Secret的安全设置：

secureSettings:
- secretName: credentials

问题分析与解决方案 #

1. 问题根源 #

该错误与install-plugins配置无关，核心原因是：

• 关联的credentials Secret中包含dev-es-snapshot-backup-feeb83405c27.json作为键，而Elasticsearch无法将该键识别为合法的安全设置项。
• 另外两个Pod运行正常，是因为它们启动时已成功加载GCS插件并完成快照仓库配置，或者本地缓存保留了正确的插件状态；故障Pod可能是新调度或重启时，Secret内容被错误解析，或插件安装临时异常，导致Elasticsearch验证安全设置时触发错误。

2. 单Pod异常原因 #

• 调度差异：故障Pod被调度到新节点，或启动时插件安装步骤临时失败，导致GCS插件未正确安装，无法识别与快照仓库相关的配置。
• 已运行Pod启动时GCS插件安装成功，可正确处理Secret内容（该JSON文件实际应为GCS快照仓库的服务账号密钥，正确方式是通过插件指定路径，而非直接作为安全设置键）。

3. 解决步骤 #

步骤1：检查Secret内容

执行命令查看credentials Secret的结构：

kubectl get secret credentials -o yaml

确认是否存在dev-es-snapshot-backup-feeb83405c27.json这个键，若存在则为错误配置。

步骤2：修正Secret与Pod配置

正确的GCS快照仓库配置需将服务账号JSON文件挂载为Volume，而非作为安全设置键：

1. 创建包含服务账号JSON的Secret：

kubectl create secret generic gcs-sa --from-file=dev-es-snapshot-backup-feeb83405c27.json=./your-service-account.json

2. 在Pod模板中添加Volume挂载：

podTemplate:
  spec:
    volumes:
    - name: gcs-sa
      secret:
        secretName: gcs-sa
    containers:
    - name: elasticsearch
      volumeMounts:
      - name: gcs-sa
        mountPath: /usr/share/elasticsearch/config/gcs
        readOnly: true

3. 在Elasticsearch配置中指定密钥文件路径：

config:
  cloud.gcs.auth.file: /usr/share/elasticsearch/config/gcs/dev-es-snapshot-backup-feeb83405c27.json

步骤3：重新调度故障Pod

删除异常Pod，让Kubernetes重新创建：

kubectl delete pod dev-es-data-hot-1

新Pod启动时会加载正确的插件与配置，避免安全设置验证错误。

步骤4：验证插件状态

进入正常Pod检查GCS插件是否安装成功：

kubectl exec -it <正常Pod名称> -- bin/elasticsearch-plugin list

确认输出包含repository-gcs。

内容的提问来源于stack exchange，提问作者Deepak