Terraform创建Databricks外部位置遇IAM角色非自承担错误求助

错误根源 #

你遇到的错误是两个核心问题叠加导致：

1. 资源创建顺序颠倒：原代码中先创建databricks_storage_credential，但此时对应的IAM角色尚未创建或完成信任策略配置，导致Databricks验证时角色不符合自承担要求
2. IAM策略传播延迟：AWS IAM的角色/策略变更需要30-60秒的全局同步时间，Databricks发起验证时可能策略还未生效

可行解决方案 #

方案1：修正资源依赖，打破循环依赖 #

由于databricks_aws_unity_catalog_assume_role_policy需要存储凭证的external_id，而存储凭证又依赖已存在的IAM角色，这里存在循环依赖。可以通过先创建临时IAM角色，再创建存储凭证，最后更新角色为正确配置的方式解决：

# 1. 先创建S3存储桶
resource "aws_s3_bucket" "external" {
  bucket        = "${local.prefix}-external"
  force_destroy = true
}

# 2. 创建临时IAM角色（初始信任策略允许基础访问，后续更新）
resource "aws_iam_role" "external_data_access_temp" {
  name = local.uc_iam_role
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action    = "sts:AssumeRole"
      },
      {
        Effect    = "Allow"
        Principal = { Service = "unitycatalog.databricks.com" }
        Action    = "sts:AssumeRole"
      }
    ]
  })
}

# 3. 创建Databricks存储凭证，关联临时IAM角色
resource "databricks_storage_credential" "external" {
  name = "${local.prefix}-external-access"
  aws_iam_role {
    role_arn = aws_iam_role.external_data_access_temp.arn
  }
}

# 4. 生成符合UC要求的信任策略（包含存储凭证的external_id）
data "databricks_aws_unity_catalog_assume_role_policy" "this" {
  aws_account_id = data.aws_caller_identity.current.account_id
  role_name      = aws_iam_role.external_data_access_temp.name
  external_id    = databricks_storage_credential.external.aws_iam_role[0].external_id
}

# 5. 生成S3桶的访问权限策略
data "databricks_aws_unity_catalog_policy" "this" {
  aws_account_id = data.aws_caller_identity.current.account_id
  bucket_name    = aws_s3_bucket.external.id
  role_name      = aws_iam_role.external_data_access_temp.name
}

# 6. 创建IAM权限策略
resource "aws_iam_policy" "external_data_access" {
  policy = data.databricks_aws_unity_catalog_policy.this.json
}

# 7. 更新IAM角色为最终配置（包含正确的信任策略和权限）
resource "aws_iam_role" "external_data_access" {
  name               = aws_iam_role.external_data_access_temp.name
  assume_role_policy = data.databricks_aws_unity_catalog_assume_role_policy.this.json
  managed_policy_arns = [aws_iam_policy.external_data_access.arn]
  
  lifecycle {
    create_before_destroy = true
  }
  depends_on = [databricks_storage_credential.external]
}

# 8. 添加等待，确保IAM配置全局生效
resource "time_sleep" "wait_for_iam_propagation" {
  create_duration = "60s" # 生产环境建议至少60秒，可根据实际调整
  depends_on = [
    aws_iam_role.external_data_access,
    aws_iam_policy.external_data_access
  ]
}

# 9. 最后创建外部位置，依赖IAM配置生效
resource "databricks_external_location" "some" {
  name            = "external"
  url             = "s3://${aws_s3_bucket.external.id}/some"
  credential_name = databricks_storage_credential.external.id
  depends_on = [time_sleep.wait_for_iam_propagation]
}

方案2：显式依赖+强制等待（适用于无循环依赖场景） #

如果你的配置不存在循环依赖，仅需调整资源顺序并添加等待：

• 确保所有IAM资源（角色、策略）在databricks_storage_credential之前创建完成
• 添加time_sleep资源等待IAM策略传播

生产环境最佳实践 #

• 模块化封装：将Unity Catalog的存储凭证、外部位置配置封装为Terraform模块，提高复用性和可维护性
• 验证IAM配置：在创建存储凭证前，用AWS CLI验证角色的自承担权限：

  aws iam simulate-principal-policy --policy-source-arn <你的角色ARN> --action-names sts:AssumeRole --resource-arns <你的角色ARN>
  

• 动态调整等待时长：根据AWS区域的网络延迟，调整time_sleep的create_duration，避免不必要的等待或等待不足
• 启用状态锁定：使用Terraform Cloud或S3状态锁定，避免多人协作时的资源冲突
• 日志排查：遇到问题时开启TF_LOG=DEBUG查看详细的资源创建日志，定位具体的配置或延迟问题

内容的提问来源于stack exchange，提问作者Olfa2