AWS DMS从Postgres RDS同步Redshift遇S3 AccessDenied问题

数据老陈

2026-8-20

问题

我们通过Terraform部署AWS DMS复制任务，将Postgres数据库全量同步至Redshift。表已按正确Schema创建，但数据滞留在DMS用作中间环节的S3桶中，未同步至Redshift。

已按Terraform文档配置dms-access-for-endpoint、dms-cloudwatch-logs-role、dms-vpc-role三个IAM角色（部署在独立栈，且被另一个正常运行的DMS实例使用），IAM配置代码如下：

data "aws_iam_policy_document" "dms_assume_role_document" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      identifiers = [
        "s3.amazonaws.com",
        "iam.amazonaws.com",
        "redshift.amazonaws.com",
        "dms.amazonaws.com",
        "redshift-serverless.amazonaws.com"
      ]
      type        = "Service"
    }
  }
}

# Database Migration Service requires the below IAM Roles to be created before
# replication instances can be created. See the DMS Documentation for
# additional information: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.html#CHAP_Security.APIRole
#  * dms-vpc-role
#  * dms-cloudwatch-logs-role
#  * dms-access-for-endpoint
resource "aws_iam_role" "dms_access_for_endpoint" {
  name                  = "dms-access-for-endpoint"
  assume_role_policy    = data.aws_iam_policy_document.dms_assume_role_document.json
  managed_policy_arns   = ["arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"]
  force_detach_policies = true
}

resource "aws_iam_role" "dms_cloudwatch_logs_role" {
  name                  = "dms-cloudwatch-logs-role"
  description           = "Allow DMS to manage CloudWatch logs."
  assume_role_policy    = data.aws_iam_policy_document.dms_assume_role_document.json
  managed_policy_arns   = ["arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole"]
  force_detach_policies = true
}

resource "aws_iam_role" "dms_vpc_role" {
  name                  = "dms-vpc-role"
  description           = "DMS IAM role for VPC permissions"
  assume_role_policy    = data.aws_iam_policy_document.dms_assume_role_document.json
  managed_policy_arns   = ["arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"]
  force_detach_policies = true
}

运行时CloudWatch日志报错：

2022-09-01T16:51:38 [SOURCE_UNLOAD   ]E:  Not retriable error: <AccessDenied> Access Denied [1001705]  (anw_retry_strategy.cpp:118)
2022-09-01T16:51:38 [SOURCE_UNLOAD   ]E:  Failed to list bucket 'dms-sandbox-redshift-intermediate-storage': error code <AccessDenied>: Access Denied [1001713]  (s3_dir_actions.cpp:105)
2022-09-01T16:51:38 [SOURCE_UNLOAD   ]E:  Failed to list bucket 'dms-sandbox-redshift-intermediate-storage' [1001713]  (s3_dir_actions.cpp:209)

S3服务器访问日志显示（已匿名化）：

<id> dms-sandbox-redshift-intermediate-storage [01/Sep/2022:15:43:32 +0000] 10.128.69.80 arn:aws:sts::<account>:assumed-role/dms-access-for-endpoint/dms-session-for-replication-engine <code> REST.GET.BUCKET - "GET /dms-sandbox-redshift-intermediate-storage?delimiter=%2F&max-keys=1000 HTTP/1.1" 403 AccessDenied 243 - 30 - "-" "aws-sdk-cpp/1.8.80/S3/Linux/4.14.276-211.499.amzn2.x86_64 x86_64 GCC/4.9.3" - <code> SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.eu-west-2.amazonaws.com TLSv1.2 -

尝试为S3桶添加桶策略但未生效，配置代码如下：

resource "aws_s3_bucket" "dms_redshift_intermediate" {
  # Prefixed with `dms-` as that's what the AmazonDMSRedshiftS3Role policy filters on
  bucket = "dms-sandbox-redshift-intermediate-storage"
}

resource "aws_s3_bucket_logging" "log_bucket" {
  bucket        = aws_s3_bucket.dms_redshift_intermediate.id
  target_bucket = aws_s3_bucket.log_bucket.id
  target_prefix = "log/"
}

resource "aws_s3_bucket" "log_bucket" {
  bucket = "${aws_s3_bucket.dms_redshift_intermediate.id}-logs"
}

resource "aws_s3_bucket_acl" "log_bucket" {
  bucket = aws_s3_bucket.log_bucket.id
  acl    = "log-delivery-write"
}

resource "aws_s3_bucket_policy" "dms_redshift_intermediate_policy" {
  bucket = aws_s3_bucket.dms_redshift_intermediate.id
  policy = data.aws_iam_policy_document.dms_redshift_intermediate_policy_document.json
}

data "aws_iam_policy_document" "dms_redshift_intermediate_policy_document" {
  statement {
    actions = [
      "s3:*"
    ]

    principals {
      identifiers = [
        "dms.amazonaws.com",
        "redshift.amazonaws.com"
      ]
      type = "Service"
    }

    resources = [
      aws_s3_bucket.dms_redshift_intermediate.arn,
      "${aws_s3_bucket.dms_redshift_intermediate.arn}/*"
    ]
  }
}

DMS可将加密CSV文件PUT至S3桶，但无法GET文件同步至Redshift，需解决该问题实现数据同步。

解决方案

1. 修正S3桶策略的授权主体

当前桶策略授权的是服务主体，但日志显示是dms-access-for-endpoint角色的会话发起请求，需直接授权该IAM角色：

data "aws_iam_policy_document" "dms_redshift_intermediate_policy_document" {
  statement {
    actions = [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ]

    principals {
      identifiers = ["arn:aws:iam::<your-account-id>:role/dms-access-for-endpoint"]
      type        = "AWS"
    }

    resources = [
      aws_s3_bucket.dms_redshift_intermediate.arn,
      "${aws_s3_bucket.dms_redshift_intermediate.arn}/*"
    ]
  }
}

2. 补充IAM角色的S3权限

AmazonDMSRedshiftS3Role托管策略可能缺少ListBucket等必要权限，为dms-access-for-endpoint添加自定义补充策略：

resource "aws_iam_policy" "dms_s3_supplemental" {
  name        = "dms-s3-supplemental-policy"
  description = "补充DMS访问S3桶的必要权限"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:ListBucket",
          "s3:GetObject"
        ]
        Effect   = "Allow"
        Resource = [
          aws_s3_bucket.dms_redshift_intermediate.arn,
          "${aws_s3_bucket.dms_redshift_intermediate.arn}/*"
        ]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "dms_s3_supplemental_attach" {
  role       = aws_iam_role.dms_access_for_endpoint.name
  policy_arn = aws_iam_policy.dms_s3_supplemental.arn
}

3. 精简IAM角色的信任策略

dms-access-for-endpoint只需要允许DMS服务扮演，移除不必要的服务主体：

data "aws_iam_policy_document" "dms_assume_role_document" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      identifiers = ["dms.amazonaws.com"]
      type        = "Service"
    }
  }
}

4. 检查KMS加密权限（若启用）

如果S3桶用KMS加密，需确保dms-access-for-endpoint拥有KMS密钥的解密权限：

resource "aws_iam_policy" "dms_kms_access" {
  name        = "dms-kms-decrypt-policy"
  description = "允许DMS解密S3桶的KMS加密文件"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = ["kms:Decrypt"]
        Effect   = "Allow"
        Resource = "<your-kms-key-arn>"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "dms_kms_attach" {
  role       = aws_iam_role.dms_access_for_endpoint.name
  policy_arn = aws_iam_policy.dms_kms_access.arn
}

5. 验证VPC端点策略（若使用VPC访问S3）

如果DMS部署在VPC内并通过S3 VPC端点访问桶，需确保VPC端点策略允许dms-access-for-endpoint执行ListBucket和GetObject操作。

内容的提问来源于stack exchange，提问作者philMarius