使用boto3的Glue API创建Apache Iceberg表报错求助

核心问题 #

你当前配置使用了Hive Avro的SerDe和输入输出格式，不符合Iceberg表的专属要求，导致Athena仍将其识别为普通Hive表，而非支持事务操作的Iceberg表。

正确配置修改点 #

需要调整以下关键配置项：

1. 替换SerDe和输入输出格式 #

Iceberg表必须使用专属的SerDe与格式：

• SerdeInfo.SerializationLibrary: org.apache.iceberg.mr.hive.HiveIcebergSerDe
• InputFormat: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
• OutputFormat: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat

2. 添加Iceberg专属参数 #

在Parameters中补充Iceberg的catalog配置与warehouse路径：

"Parameters": {
    "EXTERNAL": "TRUE",
    "avro.serialization.null.format": "NULL",
    "compressionType": "snappy",
    "iceberg.catalog.name": "glue_catalog",
    "iceberg.catalog.type": "glue",
    "warehouse": "s3://<your-warehouse-path>"
}

其中warehouse需替换为你的S3存储路径，用于存放Iceberg的元数据和数据文件。

3. 修正表的Location路径 #

Iceberg表的Location需要指向包含metadata目录的S3根路径，而非普通Avro文件的存储路径。

修改后的完整配置示例 #

{
    "Name": "<table_name_placeholder>",
    "Owner": "me",
    "StorageDescriptor": {
        "SerdeInfo": {
            "SerializationLibrary": "org.apache.iceberg.mr.hive.HiveIcebergSerDe",
            "Parameters": {
                "serialization.format": "1"
            }
        },
        "OutputFormat": "org.apache.iceberg.mr.hive.HiveIcebergOutputFormat",
        "InputFormat": "org.apache.iceberg.mr.hive.HiveIcebergInputFormat",
        "Columns": <columns_placeholder>,
        "Location": "<iceberg-table-root-location>"
    },
    "TableType": "ICEBERG",
    "Parameters": {
        "EXTERNAL": "TRUE",
        "avro.serialization.null.format": "NULL",
        "compressionType": "snappy",
        "iceberg.catalog.name": "glue_catalog",
        "iceberg.catalog.type": "glue",
        "warehouse": "s3://<your-warehouse-path>"
    }
}

验证步骤 #

1. 使用修改后的配置重新创建表
2. 在Athena中执行DESCRIBE FORMATTED <table_name>，确认Table Type为ICEBERG且参数包含Iceberg相关配置
3. 尝试执行UPDATE/INSERT/DELETE操作，检查是否仍报错

内容的提问来源于stack exchange，提问作者Steve