如何实现Glue ETL作业的跨账号Schema变更与目录同步？

问题背景 #

现有一个Glue ETL作业，配置如下：

• 数据源：关系型数据库表（通过JDBC Glue Connection连接）
• 目标：S3存储桶
• 更新选项：在数据目录中创建表，后续运行时更新Schema并添加新分区

该作业在同账号环境下运行正常，但改为跨账号部署（目标S3桶和Glue Data Catalog属于另一个账号）后，作业状态显示“成功”，且能正常向目标S3写入文件，但目标账号的Glue数据库中未创建新表。

相关日志 #

24/01/17 16:55:17 INFO HadoopDataSink: Failed to create table customer in database my_database after job run with catalogId 
com.amazonaws.services.glue.model.EntityNotFoundException: Database my_database not found. (Service: AWSGlue; Status Code: 400; Error Code: EntityNotFoundException;...

现有ETL脚本 #

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# 源 = 关系型数据库表
source_db_table = glueContext.create_dynamic_frame.from_options(
    connection_type="postgresql",
    connection_options={
        "useConnectionProperties": "true",
        "dbtable": "public.customer",
        "connectionName": "my_db_connection",
    },
    transformation_ctx="source_db_table",
)

# 目标 = S3存储桶
target_s3_bucket = glueContext.getSink(
    path="s3://my-bucket/data/customer/",
    connection_type="s3",
    updateBehavior="UPDATE_IN_DATABASE",
    partitionKeys=[],
    enableUpdateCatalog=True,
    transformation_ctx="target_s3_bucket",
)
target_s3_bucket.setCatalogInfo(
    catalogDatabase="my_database", catalogTableName="customer"
)
target_s3_bucket.setFormat("glueparquet", compression="snappy")
target_s3_bucket.writeFrame(source_db_table)
job.commit()

现有Glue Data Catalog资源策略 #

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/my-role"
      },
      "Action": [
        "glue:CreateTable",
        "glue:DeleteTable",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:UpdateTable"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:222222222222:catalog",
        "arn:aws:glue:us-east-1:222222222222:database/my_database",
        "arn:aws:glue:us-east-1:222222222222:table/my_database/*"
      ]
    }
  ]
}

问题分析与解决步骤 #

核心原因 #

日志显示Database my_database not found，本质是跨账号场景下，Glue作业默认只会查询当前作业所在账号的Data Catalog，无法定位到目标账号的数据库。另外现有资源策略缺少必要的数据库查询权限，导致作业无法验证目标数据库的存在性。

具体修复措施 #

1. 修改ETL脚本，指定目标账号的Catalog ID
   在setCatalogInfo方法中添加catalogId参数，明确指向目标账号的Data Catalog：

   target_s3_bucket.setCatalogInfo(
       catalogDatabase="my_database", 
       catalogTableName="customer",
       catalogId="222222222222"  # 替换为目标账号的ID
   )
   

2. 补充Glue Data Catalog资源策略的权限
   现有策略缺少glue:GetDatabase权限，作业需要该权限验证目标数据库是否存在，更新后的策略如下：

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::111111111111:role/my-role"
         },
         "Action": [
           "glue:CreateTable",
           "glue:DeleteTable",
           "glue:GetPartitions",
           "glue:GetTable",
           "glue:UpdateTable",
           "glue:GetDatabase"  # 新增该权限
         ],
         "Resource": [
           "arn:aws:glue:us-east-1:222222222222:catalog",
           "arn:aws:glue:us-east-1:222222222222:database/my_database",
           "arn:aws:glue:us-east-1:222222222222:table/my_database/*"
         ]
       }
     ]
   }
   

3. 确认目标账号的数据库已存在
   确保目标账号（222222222222）的Glue中已创建my_database，若不存在需先创建该数据库。

4. 验证作业角色的权限
   确保作业执行角色（arn:aws:iam::111111111111:role/my-role）的IAM策略中，允许调用目标账号Glue的相关API操作，可添加如下IAM权限语句：

   {
     "Effect": "Allow",
     "Action": [
       "glue:GetDatabase",
       "glue:CreateTable",
       "glue:UpdateTable"
     ],
     "Resource": [
       "arn:aws:glue:us-east-1:222222222222:catalog",
       "arn:aws:glue:us-east-1:222222222222:database/my_database",
       "arn:aws:glue:us-east-1:222222222222:table/my_database/*"
     ]
   }
   

内容的提问来源于stack exchange，提问作者srk