EC2实例通过S3网关端点无法访问Amazon Linux源安装docker求助

故障背景 #

ASG中的EC2实例初始化时无法安装Docker，经初步定位故障与S3端点相关。实例部署在同时包含公有、私有子网的VPC内，路由、EIP/NAT配置均已完成：私有子网实例0.0.0.0/0出站流量路由到公有子网NAT网关；公有子网NACL允许互联网双向通行，私有子网NACL允许与公有子网双向流量、到互联网的出站流量，以及S3 CIDR段双向流量，整体安全策略配置严格。

已确认配置 #

• VPC已开启DNS解析和DNS主机名功能
• 已考虑NACL无状态特性，为S3亚马逊IP段配置了临时端口范围的入站、出站规则，同时已放行公、私有子网间的流量
• 私有路由表中已存在S3端点的对应路由
• 已确认故障由S3端点导致：删除S3端点并放开NACL限制后，yum update和Docker安装均可正常执行。本次目标是通过VPC网关端点保障私有子网安全性，不接受放开NACL限制的解决方案
• 排除预制预装Docker的AMI方案，本次为网络配置练习，需要根因解决问题而非绕开
• 其余VPC端点运行正常：Auto Scaling接口端点可正常执行扩缩容策略，SSM接口端点支持正常使用Session Manager，ECR端点配合S3网关端点可正常拉取镜像（镜像层存放在S3中，功能验证正常），说明S3网关端点可正常访问ECR镜像层，但无法访问amazon-linux-extras源
• 实例关联的安全组无问题，实例默认带有全量出站规则
• 已尝试放宽S3端点权限策略、确认区域配置正确，仍未解决，判断问题大概率出在S3网关策略上

现有CloudFormation S3端点配置 #

S3Endpoint:
  Type: 'AWS::EC2::VPCEndpoint'
  Properties:
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal: '*'
          Action:
            - 's3:GetObject'
          Resource: 
            - 'arn:aws:s3:::prod-ap-southeast-2-starport-layer-bucket/*'
            - 'arn:aws:s3:::packages.*.amazonaws.com/*'
            - 'arn:aws:s3:::repo.*.amazonaws.com/*'
            - 'arn:aws:s3:::amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/*'
            - 'arn:aws:s3:::amazonlinux.*.amazonaws.com/*'
            - 'arn:aws:s3:::*.amazonaws.com'
            - 'arn:aws:s3:::*.amazonaws.com/*'
            - 'arn:aws:s3:::*.ap-southeast-2.amazonaws.com/*'
            - 'arn:aws:s3:::*.ap-southeast-2.amazonaws.com/'
            - 'arn:aws:s3:::*repos.ap-southeast-2-.amazonaws.com'
            - 'arn:aws:s3:::*repos.ap-southeast-2.amazonaws.com/*'
            - 'arn:aws:s3:::repo.ap-southeast-2-.amazonaws.com'
            - 'arn:aws:s3:::repo.ap-southeast-2.amazonaws.com/*'
    RouteTableIds:
      - !Ref PrivateRouteTableA
      - !Ref PrivateRouteTableB   
    ServiceName: !Sub 'com.amazonaws.${AWS::Region}.s3'
    VpcId: !Ref BasicVpc
    VpcEndpointType: Gateway

上述配置中第一条规则是ECR接口端点从S3拉取镜像层的必要配置，其余规则均为尝试访问amazon-linux-extras源添加的配置。

故障复现现象 #

通过SSM端点使用Session Manager连接到实例可复现初始化故障：

1. 执行yum install docker -y报错如下：

Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Could not retrieve mirrorlist https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/core/latest/x86_64/mirror.list error was
14: HTTPS Error 403 - Forbidden
已配置的某一存储库故障，yum无足够缓存数据继续运行，可通过以下方案尝试修复：

1. 联系存储库上游提供商修复问题
2. 重新配置存储库的baseurl等参数，指向可用的上游地址
3. 临时禁用故障存储库执行命令：yum --disablerepo= ...
4. 永久禁用故障存储库：yum-config-manager --disable 或 subscription-manager repos --disable
5. 配置存储库不可用时自动跳过：yum-config-manager --save --setopt=.skip_if_unavailable=true
   Cannot find a valid baseurl for repo: amzn2-core/2/x86_64

2. 执行amazon-linux-extras install docker报错如下：

Catalog is not reachable. Try again later.
catalogs at https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/extras-catalog-x86_64-v2.json, https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/extras-catalog-x86_64.json
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/amazon_linux_extras/software_catalog.py", line 131, in fetch_new_catalog
request = urlopen(url)
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden

根因说明 #

1. S3端点策略中存在无效ARN匹配规则：部分规则错误将S3访问域名作为桶名填写、存在多余符号，无法匹配到真实的Amazon Linux 2 yum源桶；
2. 桶级权限缺失：部分yum操作需要对仓库桶执行s3:ListBucket权限，原有策略仅配置了对象级的s3:GetObject权限，未覆盖桶级操作。

修复后的S3端点策略 #

替换原有PolicyDocument为以下配置即可：

PolicyDocument:
  Version: 2012-10-17
  Statement:
    - Effect: Allow
      Principal: '*'
      Action:
        - 's3:GetObject'
        - 's3:ListBucket'
      Resource: 
        - 'arn:aws:s3:::prod-ap-southeast-2-starport-layer-bucket'
        - 'arn:aws:s3:::prod-ap-southeast-2-starport-layer-bucket/*'
        - 'arn:aws:s3:::amazonlinux-2-repos-ap-southeast-2'
        - 'arn:aws:s3:::amazonlinux-2-repos-ap-southeast-2/*'
        - 'arn:aws:s3:::packages.ap-southeast-2.amazonaws.com'
        - 'arn:aws:s3:::packages.ap-southeast-2.amazonaws.com/*'
        - 'arn:aws:s3:::repo.ap-southeast-2.amazonaws.com'
        - 'arn:aws:s3:::repo.ap-southeast-2.amazonaws.com/*'

验证步骤 #

修复部署后登录实例执行以下命令验证：

1. 清理yum缓存：yum clean all
2. 重建yum缓存：yum makecache
3. 执行Docker安装：amazon-linux-extras install docker -y

内容的提问来源于stack exchange，提问作者GorginZ