使用Elasticsearch Operator配置GCP存储库失败

问题描述 #

通过Elastic Operator部署3个Elasticsearch节点，配置GCS自动快照时，按步骤操作：将GCS服务账号密钥JSON压缩后创建名为gcs.client.default.credentials_file的无扩展名文件，存入Kubernetes Secretgcs-credentials，并在Elasticsearch CR的spec.secureSettings.secretName中指定该Secret。但Elasticsearch启动时抛出致命异常，关键错误日志如下：

{"@timestamp":"2022-12-26T18:45:40.037Z", "log.level":"ERROR", "message":"Elasticsearch启动时发生致命异常", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-cluster-es-node-1","elasticsearch.cluster.name":"elasticsearch-cluster","error.type":"java.lang.IllegalStateException","error.message":"加载插件类[org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]失败","error.stack_trace":"java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:482)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:290)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:159)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:634)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:406)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.node.Node.<init>(Node.java:316)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)\n\tat org.elasticsearch.server@8.5.0/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:600)\n\t... 9 more\nCaused by: java.lang.IllegalArgumentException: 从[gcs.client.default.credentials_file]加载GCS客户端凭证失败\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:265)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.getClientSettings(GoogleCloudStorageClientSettings.java:221)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.load(GoogleCloudStorageClientSettings.java:209)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.reload(GoogleCloudStoragePlugin.java:88)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.<init>(GoogleCloudStoragePlugin.java:36)\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)\n\t... 12 more\nCaused by: java.io.IOException: 无效的PKCS#8数据。\n\tat com.google.auth.oauth2.ServiceAccountCredentials.privateKeyFromPkcs8(ServiceAccountCredentials.java:496)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromPkcs8(ServiceAccountCredentials.java:474)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromJson(ServiceAccountCredentials.java:212)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:548)\n\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:520)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.lambda$loadCredential$13(GoogleCloudStorageClientSettings.java:257)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:33)\n\tat org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:256)\n\t... 17 more\n"}
ERROR: Elasticsearch未正常退出 - 请查看日志文件/usr/share/elasticsearch/logs/elasticsearch-cluster.log

解决方案 #

1. 废弃压缩操作，使用原始GCS密钥JSON #

不要对服务账号密钥JSON文件进行压缩，直接使用Google Cloud控制台下载的原始JSON文件——该文件包含完整的PKCS#8格式私钥，压缩会破坏文件结构导致解析失败。

2. 重新创建Kubernetes Secret #

使用kubectl直接从原始JSON文件生成Secret，避免手动创建文件时的格式错误：

kubectl create secret generic gcs-credentials --from-file=gcs.client.default.credentials_file=/path/to/your/original-service-account-key.json

3. 验证Secret内容正确性 #

执行以下命令解码Secret中的凭证内容，确认与原始JSON完全一致：

kubectl get secret gcs-credentials -o jsonpath='{.data.gcs\.client\.default\.credentials_file}' | base64 --decode

检查输出的JSON中private_key字段是否以-----BEGIN PRIVATE KEY-----开头，-----END PRIVATE KEY-----结尾，且中间内容无缺失或乱码。

4. 确认Elasticsearch CR配置无误 #

检查Elasticsearch自定义资源的secureSettings配置，确保指向正确的Secret名称：

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-cluster
spec:
  version: 8.5.0
  secureSettings:
  - secretName: gcs-credentials
  # 其他集群配置（节点数、资源等）

5. 触发集群滚动重启 #

修改Secret或CR配置后，执行以下命令重启Elasticsearch集群，使新配置生效：

kubectl rollout restart statefulset/elasticsearch-cluster-es-default

错误原因 #

核心问题是压缩后的JSON密钥文件破坏了PKCS#8私钥的格式，GCS插件解析凭证时无法识别损坏的私钥数据，导致插件加载失败，最终引发Elasticsearch启动异常。

内容的提问来源于stack exchange，提问作者Mahmoud Yasser