跨区域还原快照至AWS ES 6.2时S3返回500错误

最近我尝试把存储在us-east-1区域S3桶里的Elasticsearch v5.3快照，跨账户、跨区域还原到us-west-2区域的AWS ES 6.2集群，过程中碰到了挺头疼的S3 301重定向错误，跟大家梳理下整个过程：

初始快照仓库配置 #

首先在目标ES集群上注册了快照仓库，配置如下：

{ 
  "type": "s3", 
  "settings": { 
    "bucket": "valid-bucket-name", 
    "server_side_encryption": "true", 
    "endpoint": "s3.amazonaws.com", 
    "region" : "us-east-1", 
    "role_arn": "valid-role" 
  } 
}

注册后返回了成功响应：

{ "acknowledged": true }

遇到的301错误 #

本以为配置没问题，结果尝试还原特定快照时，直接收到了S3的301永久重定向错误：

{ 
  "error": { 
    "root_cause": [ 
      { 
        "type": "amazon_s3_exception", 
        "reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: D72EFE8A89F76F57; S3 Extended Request ID: in03KW452re297MDp3GQQRFjJhMRXeP4md+FU99CHZ7D4TQKz8PBuSZKoO3+IFd+wAxNApztG5Y=)" 
      } 
    ], 
    "type": "amazon_s3_exception", 
    "reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: D72EFE8A89F76F57; S3 Extended Request ID: in03KW452re297MDp3GQQRFjJhMRXeP4md+FU99CHZ7D4TQKz8PBuSZKoO3+IFd+wAxNApztG5Y=)" 
  }, 
  "status": 500 
}

奇怪的是，仓库明明已经配置了us-east-1区域，错误信息完全没有给出有效的调整方向。后来按照@Michael-sqlbot的建议修改了endpoint，但问题依然存在，报错变成了：

{ 
  "error": { 
    "root_cause": [ 
      { 
        "type": "amazon_s3_exception", 
        "reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: CC1853D0EF68B5F7; S3 Extended Request ID: nrbWmI3OiPLdrMcRT6FiOHJineYv6clmSf+GcXtBBwKSzfIEV2gmMZjWEDtyCIRQUg+dM/Vmawg=)" 
      } 
    ], 
    "type": "blob_store_exception", 
    "reason": "Failed to check if blob [master.dat-temp] exists", 
    "caused_by": { 
      "type": "amazon_s3_exception", 
      "reason": "amazon_s3_exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: CC1853D0EF68B5F7; S3 Extended Request ID: nrbWmI3OiPLdrMcRT6FiOHJineYv6clmSf+GcXtBBwKSzfIEV2gmMZjWEDtyCIRQUg+dM/Vmawg=)" 
    } 
  }, 
  "status": 500 
}

最终解决办法 #

经过反复排查验证，最终确认这是s3-snapshot-plugin的区域/端点兼容性问题。我在us-east-1区域创建了临时ES集群，然后在这个同区域的集群上还原快照，成功解决了问题。

内容的提问来源于stack exchange，提问作者Nithin Chandy