如何让AWS EMR访问Lake Formation资源链接表？

问题背景 #

持有来自其他AWS账号的Lake Formation资源链接数据库表，在Athena中可正常查询，但EMR的PySpark环境下无法读取数据。已将EMR服务及实例IAM角色设置为Lake Formation管理员，仍存在权限传递失效问题。该表同时为Iceberg表，暂不确定是否影响权限逻辑。

当前Spark配置 #

{
  "Classification": "spark-defaults",
  "Properties": {
    "spark.sql.catalog.aws_glue": "org.apache.iceberg.spark.SparkCatalog",
    "spark.sql.catalog.aws_glue.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",
    "spark.sql.catalog.aws_glue.glue.lakeformation.enabled": "true",
    "spark.sql.catalog.aws_glue.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "spark.sql.catalog.aws_glue.lakeformation-enabled": "true",
    "spark.sql.defaultCatalog": "aws_glue"
  }
}

测试步骤与现象 #

1. 目录表列表可正常加载：

# List tables first to verify access
logger.info("Verifying table access...")
tables = spark_session.sql(f"SHOW TABLES FROM {catalog_name}.{db_name}").collect()
logger.info(f"Available tables: {[t.tableName for t in tables]}")

日志输出：

Verifying table access...
2025-01-16 05:42:06 INFO     Available tables: ['account', 'activitydefinition',...

2. 多种查询方式均触发权限错误：

# First try a simple count to verify access
logger.info("\nAttempting count query")
try:
    count_df = spark_session.sql(f"""
        SELECT *
        FROM {catalog_name}.{db_name}.{table_name}
    """)
    count_df.show()
except Exception as e:
    logger.error(f"Count query failed: {str(e)}")
# Try reading with minimal options
logger.info("\nAttempting main query")
try:
    df = (
        spark_session.read.format("iceberg")
        .option("lakeformation-enabled", "true")
        .option("read-identity-based-auth", "true")
        .table(f"{db_name}.{table_name}")
        .select("id", "identifier")
    )
    logger.info("Successfully created DataFrame")
    df.printSchema()
    return df
except Exception as e:
    logger.error(f"Main query failed: {str(e)}")
    # One final attempt with SQL
    logger.info("\nTrying final SQL approach")
    df = spark_session.sql(f"""
        SELECT t.*
        FROM {catalog_name}.{db_name}.{table_name} t
    """)
    return df

错误信息（已翻译） #

查询数据失败：调用o149.sql时出错。错误类型：software.amazon.awssdk.services.s3.model.S3Exception：访问被拒绝（服务：S3，状态码：403，请求ID：FRKVTCMCWA771WS7，扩展请求ID：rH0oJbyJm6IBsmZCMDlOZzbjh5hxBE5oU31zXxnxolomK4a+c4txq7iTV4I7WDsgC32qXMnEAUw=）
堆栈跟踪片段：
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
...

解决建议 #

• 补全S3底层权限：Lake Formation管理员权限不自动包含S3对象访问权限。跨账号场景下，需在存储桶所属账号的S3策略中为EMR实例角色添加s3:GetObject、s3:ListBucket权限；或通过Lake Formation给EMR角色授予目标数据位置的访问权限，而非仅表权限。
• 统一Iceberg配置：当前Spark配置中重复设置了Lake Formation启用参数，保留spark.sql.catalog.aws_glue.lakeformation-enabled = true即可。同时添加spark.sql.catalog.aws_glue.warehouse指定Iceberg仓库路径，确保该路径的Lake Formation权限已覆盖EMR角色。
• 完善跨账号权限传递：资源链接表所属账号需将表的SELECT权限和底层数据位置权限授予EMR所在账号的角色；EMR所在账号需将这些权限进一步传递给EMR实例角色。
• 验证基础S3访问：在EMR节点执行aws s3 ls <目标S3路径>测试直接访问，排除Spark/Iceberg配置外的权限问题。
• 检查角色信任策略：确认EMR服务角色和实例角色的信任策略允许Lake Formation服务调用，且角色已正确关联Lake Formation管理员权限。

内容的提问来源于stack exchange，提问作者vfrank66