ECS Fargate通过Secret Manager连接RDS遇Access Denied问题排查

问题概述 #

在AWS ECS上通过Jenkins+ECR部署Laravel应用，新任务创建成功，但连接RDS数据库时触发「Access Denied」错误。经排查，应用未正确从Secret Manager获取数据库凭证， fallback到config/database.php中的默认用户forge，最终导致权限拒绝。

部署文件参考 #

Jenkinsfile #

pipeline {
    agent any

    environment {
        AWS_ACCOUNT_ID="794664785634"
        AWS_DEFAULT_REGION="us-east-1"
        IMAGE_REPO_NAME="product-mangement"
        IMAGE_TAG="${BUILD_NUMBER}"
        REPOSITORY_URI = "794664785634.dkr.ecr.us-east-1.amazonaws.com/product-mangement"
        ECS_CLUSTER = "product-mangement"
        ECS_SERVICE = "product-mangement"
    }

    stages {
        stage('Checkout Latest Source') {
            steps {
                git branch: 'master',
                url: 'https://github.com/jhon-123/product-mangement',
                credentialsId: 'jenkins_pta'
            }
        }
        stage('Logging into AWS ECR') {
            steps {
                script {
                    sh """aws ecr get-login-password --region ${AWS_DEFAULT_REGION} | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"""
                }
                 
            }
        }
        
        // Building Docker images
        stage('Building image') {
            steps{
                script {
                    dockerImage = docker.build "${IMAGE_REPO_NAME}:${IMAGE_TAG}"
                }
            }
        }
   
        // Uploading Docker images into AWS ECR
        stage('Pushing to ECR') {
            steps{  
                script {
                    sh """docker tag ${IMAGE_REPO_NAME}:${IMAGE_TAG} ${REPOSITORY_URI}:$IMAGE_TAG"""
                    sh """docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}:${IMAGE_TAG}"""
                }
            }
        }

        stage('Deploy to ECS') {
            steps {
                sh "aws ecs update-service --cluster ${ECS_CLUSTER} --service ${ECS_SERVICE} --force-new-deployment"
            }
        }
    }
}

Dockerfile #

# Use the official PHP image as a base
FROM php:8.1-fpm

ENV COMPOSER_ALLOW_SUPERUSER 1

# Arguments defined in docker-compose.yml
ARG user
ARG uid

USER root

# Install system dependencies
RUN apt-get update && apt-get install -y \
    git \
    curl \
    libpng-dev \
    libonig-dev \
    libxml2-dev \
    zip \
    unzip

# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Install PHP extensions
RUN docker-php-ext-install pdo_mysql mbstring exif pcntl bcmath gd

# Get latest Composer
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer

# Create system user to run Composer and Artisan Commands
# RUN useradd -G www-data,root -u $uid -d /home/$user $user
# RUN mkdir -p /home/$user/.composer && \
  # chown -R $user:$user /home/$user

# Set the working directory
WORKDIR /var/www

# Copy the project files into the container
COPY . /var/www

# Copy .env.example to .env
COPY .env.prod .env

# Install Composer dependencies
RUN composer install

# Cache configuration
RUN php artisan config:clear
RUN php artisan config:cache

# Generate Laravel application key
RUN php artisan key:generate

# Copy the start script into the container
COPY script.sh /var/www/script.sh

# Make the script executable
RUN chmod +x /var/www/script.sh

# Expose port 8000
EXPOSE 8000

# show message
RUN echo "ehllo"

# Run the start script as the CMD
CMD ["/var/www/script.sh"]

script.sh #

#!/bin/sh

# Run Laravel migrations
php artisan migrate

# Seed Database
php artisan db:seed
echo "seeded successfully"

# Start the Laravel application
php artisan serve --host=0.0.0.0 --port=8000

已尝试操作 #

• 从自定义VPC切换至默认VPC，问题依旧
• 为任务角色附加以下权限：AmazonEC2ContainerServiceRole、AmazonECS_FullAccess、AmazonECSTaskExecutionRolePolicy、AmazonRDSDataFullAccess、AmazonRDSFullAccess
• 将RDS用户名和密码存入Secret Manager，在ECS任务定义环境变量中通过ValueFrom引用ARN：

  arn:aws:secretsmanager:us-east-1:794664117291:secret:product-management-secrets-83qahw::username:
  arn:aws:secretsmanager:us-east-1:794664117291:secret:product-management-secrets-83qahw::password:
  

  同时将数据库引擎、主机、端口信息写入.env.prod并复制到容器中

故障排查与解决办法 #

1. 修正ECS角色权限配置 #

从Secret Manager拉取凭证由任务执行角色负责，而非任务角色。检查任务执行角色是否拥有secretsmanager:GetSecretValue权限：

• 若AmazonECSTaskExecutionRolePolicy未生效，添加自定义策略明确允许访问目标Secret：

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": "secretsmanager:GetSecretValue",
              "Resource": "arn:aws:secretsmanager:us-east-1:794664117291:secret:product-management-secrets-83qahw"
          }
      ]
  }
  

2. 修正Secret Manager ARN引用格式 #

你当前的ARN格式错误，正确的引用方式为：

• 假设Secret存储的是JSON格式（{"username":"xxx","password":"xxx"}），ARN应写为：

  arn:aws:secretsmanager:us-east-1:794664117291:secret:product-management-secrets-83qahw:username::
  arn:aws:secretsmanager:us-east-1:794664117291:secret:product-management-secrets-83qahw:password::
  

  格式规则：ARN:密钥名称::，而非ARN::密钥名称:

3. 确保配置缓存读取最新环境变量 #

Dockerfile中提前执行config:cache会缓存构建时的.env配置，导致ECS注入的环境变量无法生效：

• 修改script.sh，在启动前重新缓存配置：

  #!/bin/sh
  
  # 清除旧缓存并重新缓存配置，读取最新环境变量
  php artisan config:clear
  php artisan config:cache
  
  php artisan migrate
  php artisan db:seed
  echo "seeded successfully"
  
  php artisan serve --host=0.0.0.0 --port=8000
  

• 或在.env.prod中使用环境变量占位符，让容器启动时自动替换：

  DB_USERNAME=${DB_USERNAME}
  DB_PASSWORD=${DB_PASSWORD}
  

4. 验证网络与安全组配置 #

• 确认ECS任务安全组允许出站访问RDS的对应端口（如MySQL的3306）
• 确认RDS安全组允许ECS任务安全组的入站连接
• 在ECS任务中执行telnet <RDS_HOST> <PORT>或nc -zv <RDS_HOST> <PORT>验证网络连通性

5. 确认Secret与RDS凭证匹配 #

• 手动从Secret Manager获取凭证，确认用户名、密码与RDS实例的实际凭证一致
• 确保RDS用户拥有目标数据库的访问权限，执行SQL：

  GRANT ALL ON your_database_name.* TO 'your_db_user'@'%';
  FLUSH PRIVILEGES;
  

6. 查看日志定位问题 #

• 查看ECS任务的stdout/stderr日志，确认环境变量是否注入成功
• 查看Laravel日志（storage/logs/laravel.log），获取详细的数据库连接错误信息，确认是否使用了默认用户forge

内容的提问来源于stack exchange，提问作者Spiral