无法通过opensearch-keystore读取密钥，S3快照仓库创建失败

问题现象 #

创建S3兼容的OpenSearch快照仓库时触发repository_verification_exception报错，底层提示连接S3端点超时；但在请求中手动指定access_key和secret_key时，仓库创建请求可成功执行。

报错信息 #

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[search] path [my/snapshot/directory] is not accessible on cluster-manager node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[search] path [my/snapshot/directory] is not accessible on cluster-manager node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [my/snapshot/directory/tests-BjniINtaTreGkZPpQu4zRw/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Read timed out"
        }
      }
    }
  },
  "status": 500
}

已执行操作 #

• 无密钥的创建请求（失败）：

PUT /_snapshot/search
{
  "type": "s3",
  "settings": {
    "bucket": "search-backup",
    "base_path": "my/snapshot/directory",
    "endpoint": "https://my-endpoint"
  }
}

• 手动指定密钥的创建请求（成功）：

PUT /_snapshot/search
{
  "type": "s3",
  "settings": {
    "bucket": "search-backup",
    "base_path": "my/snapshot/directory",
    "endpoint": "https://my-endpoint",
    "access_key": "xxx",
    "secret_key": "xxx"
  }
}

• 通过opensearch-keystore添加默认客户端密钥：

echo {{ backup_access_key }} | podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore add --force --stdin s3.client.default.access_key

echo {{ backup_secret_key }} | podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore add --force --stdin s3.client.default.secret_key 

• 确认密钥已存在于keystore中：

podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

缺失的配置/操作 #

1. 重启OpenSearch服务
   opensearch-keystore中新增的密钥需要重启OpenSearch进程才能加载生效。你仅完成了密钥添加操作，但未重启服务，导致集群无法读取到keystore中的密钥配置，进而触发连接认证失败和超时。

2. 验证keystore文件权限
   确保cluster-manager节点上的opensearch.keystore文件权限为opensearch:opensearch，避免进程无法读取密钥：

podman exec opensearch ls -l /usr/share/opensearch/config/opensearch.keystore

3. 确认客户端配置匹配（可选）
   若快照仓库未指定client参数，默认使用s3.client.default客户端，你的密钥配置项是正确的；如果后续自定义了客户端名称，需同步修改keystore中的密钥命名（如s3.client.my_custom_client.access_key）。

内容的提问来源于stack exchange，提问作者CJW