启用SSE-S3加密的Minio Bucket复制权限不足问题排查请求

问题描述 #

我有两个通过MinIO/KES与HashiCorp Vault启用SSE-S3加密的MinIO实例。为public桶配置跨实例复制后，初期运行约1分钟正常，随后复制停止。未启用SSE-S3时复制功能正常。

错误信息 #

目标MinIO实例报错 #

DeploymentID: 89be6e21-8a84-4e91-9689-87c9b5fa083a
not authorized: insufficient permissions (kes.Error)

KES服务器报错 #

{"time":"2023-07-07T11:14:48.583146888Z","request":{"ip":"192.177.0.2","path":"/v1/status"},"response":{"code":403,"time":16000}}

Docker Compose配置 #

version: '3.8'

networks:
  storage-network:
    ipam:
      driver: default
      config:
        - subnet: 192.177.0.0/25
          gateway: 192.177.0.1

services:
  app-minio1:
    image: minio/minio
    restart: always
    container_name: app-minio
    depends_on:
      - minio-kes1
    volumes:
      - /mnt/minio:/data
      - ./Minio/env/minio:/etc/default/minio/minio
      - ./Minio/KES/certs:/etc/default/minio/certs
    environment:
      - 'MINIO_ROOT_USER=${MINIO_USERNAME}'
      - 'MINIO_ROOT_PASSWORD=${MINIO_PASSWORD}'
      - MINIO_CONFIG_ENV_FILE=/etc/default/minio/minio
    ports:
      - 9001:9001
      - 9000:9000
    command: server --console-address ":9001" /data
    networks:
      storage-network:
        ipv4_address: 192.177.0.2
  valut1:
    image: vault:1.13.3
    restart: always
    container_name: vault
    cap_add:
      - IPC_LOCK
    environment:
      - 'VAULT_LOCAL_CONFIG={"backend": {"file": {"path": "/vault/file"}}, "default_lease_ttl": "168h", "max_lease_ttl": "720h"}'
      -  VAULT_ADDR=http://127.0.0.1:8200
    ports:
      - '8200:8200'
    volumes:
      - /vault:/vault
      - ./Vault:/etc/vault
    command: server -config /etc/vault/main.hcl
    networks:
      storage-network:
        ipv4_address: 192.177.0.3
  minio-kes1:
    image: minio/kes
    container_name: minio-kes
    restart: always
    volumes:
      - ./Minio/KES:/root/.kes
    environment:
    - KES_SERVER=https://0.0.0.0:7373
    - KES_CLIENT_CERT=/root/.kes/certs/minio-kes.cert
    - KES_CLIENT_KEY=/root/.kes/certs/minio-kes.key
    ports:
      - 7373:7373
    cap_add:
      - IPC_LOCK
    command: server --config=/root/.kes/config/config.yaml --auth=off 
    networks:
      storage-network:
        ipv4_address: 192.177.0.4
  app-minio3:
    image: minio/minio
    restart: always
    container_name: app-minio3
    depends_on:
      - minio-kes3
    volumes:
      - /mnt/minio3:/data
      - ./Minio3/env/minio:/etc/default/minio/minio
      - ./Minio3/KES/certs:/etc/default/minio/certs
    environment:
      - 'MINIO_ROOT_USER=${MINIO_USERNAME}'
      - 'MINIO_ROOT_PASSWORD=${MINIO_PASSWORD}'
      - MINIO_CONFIG_ENV_FILE=/etc/default/minio/minio
    ports:
      - 9005:9001
      - 9004:9000
    command: server --console-address ":9001" /data
    networks:
      storage-network:
        ipv4_address: 192.177.0.8
  valut3:
    image: vault:1.13.3
    restart: always
    container_name: vault3
    cap_add:
      - IPC_LOCK
    environment:
      - 'VAULT_LOCAL_CONFIG={"backend": {"file": {"path": "/vault/file"}}, "default_lease_ttl": "168h", "max_lease_ttl": "720h"}'
      -  VAULT_ADDR=http://127.0.0.1:8200
    ports:
      - '8203:8200'
    volumes:
      - /vault3:/vault
      - ./Vault:/etc/vault
    command: server -config /etc/vault/main.hcl
    networks:
      storage-network:
        ipv4_address: 192.177.0.9
  minio-kes3:
    image: minio/kes
    container_name: minio-kes3
    restart: always
    volumes:
      - ./Minio3/KES:/root/.kes
    environment:
    - KES_SERVER=https://0.0.0.0:7373
    - KES_CLIENT_CERT=/root/.kes/certs/minio-kes.cert
    - KES_CLIENT_KEY=/root/.kes/certs/minio-kes.key
    ports:
      - 7375:7373
    cap_add:
      - IPC_LOCK
    command: server --config=/root/.kes/config/config.yaml --auth=off 
    networks:
      storage-network:
        ipv4_address: 192.177.0.10

复制配置 #

排查步骤 #

1. 检查KES服务器策略配置 #

尽管启动KES时添加了--auth=off，仍需确认config.yaml中的策略是否允许目标MinIO实例（192.177.0.2）访问必要的KES API。确保策略包含以下权限：

• /v1/key/create/*
• /v1/key/generate/*
• /v1/key/decrypt/*
• /v1/status（解决KES返回的403错误）

示例策略配置：

policy:
  my-policy:
    allow:
    - /v1/key/create/*
    - /v1/key/generate/*
    - /v1/key/decrypt/*
    - /v1/status
    identities:
    - "*"  # 或指定MinIO实例的客户端证书指纹

2. 验证MinIO与KES的证书信任关系 #

• 确认两个MinIO实例的客户端证书（minio-kes.cert）已被对应KES服务器信任，检查KES配置中的root证书是否包含MinIO证书的CA。
• 验证证书有效期，避免证书过期导致权限失效。

3. 检查Vault密钥租赁周期 #

Vault配置的default_lease_ttl为168h，但KES可能在密钥租赁到期后无法自动续期。查看Vault审计日志，确认KES是否能正常续期密钥租赁；若续期失败，需确保KES的Vault令牌有足够权限执行renew操作。

4. 检查MinIO复制的加密配置 #

• 确保复制配置中勾选了「加密对象复制」选项，目标MinIO实例需能访问源实例使用的加密密钥，或配置了密钥复制策略。
• 验证源MinIO实例是否将加密密钥元数据传递给目标实例，目标实例能否通过自身KES服务器解密或获取密钥。

5. 查看MinIO与KES的详细日志 #

• 启用MinIO调试日志：在启动命令中添加--debug参数，查看复制过程中与KES交互的详细错误信息。
• 查看KES服务器完整日志，除/v1/status的403外，是否存在其他API调用的权限错误。

6. 确认网络连通性与防火墙 #

尽管初期复制正常，仍需检查是否存在间歇性网络阻断，或防火墙规则限制了MinIO与KES之间的持续连接。

内容的提问来源于stack exchange，提问作者Amir Almian