AWS Glue Iceberg未使用Hive却报‘无法连接Hive元存储’问题排查

我按照AWS官方教程创建AWS Glue Job测试Apache Iceberg，未使用Hive，想使用AWS Glue Catalog，代码里也没有Hive相关配置，但Glue仍尝试连接Hive元存储，报错“Failed to connect to Hive Metastore”。

我的代码 #

from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from awsglue.context import GlueContext

DB_NAME='default'
CATALOG_NAME="glue_catalog"  #The AWS Glue Data Catalog is pre-configured for use by the Spark libraries as glue_catalog.  
TABLE_NAME = "table1"

conf = (SparkConf().setAppName("Spark Test") 
    .set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .set(f"spark.sql.catalog.{CATALOG_NAME}", "org.apache.iceberg.spark.SparkCatalog")  #this does not enable iceberg support?
    .set(f"spark.sql.catalog.{CATALOG_NAME}.warehouse", "s3://<your-warehouse-dir>")
    .set(f"spark.sql.catalog.{CATALOG_NAME}.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") 
    .set(f"spark.sql.catalog.{CATALOG_NAME}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    .set(f"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")  
    )


#create spark runner
spark = ( SparkSession 
    .builder 
   .appName("Python Spark Iceberg example") 
   .config(conf=conf) 
   .getOrCreate() 
        )
        
glueContext = GlueContext(spark.sparkContext.getOrCreate())  #not sure what this is for but I tried it, no difference seen.

#directly from https://docs.aws.amazon.com/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-spark.html
glueContext.sql(f"""
    CREATE TABLE IF NOT EXISTS {CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}_nopartitions (
        c_customer_sk             int,
        c_customer_id             string,
        c_first_name              string,
        c_last_name               string,
        c_birth_country           string,
        c_email_address           string)
    USING iceberg
    OPTIONS ('format-version'='2')
""")

#### ERROR "Failed to connect to Hive Metastore"  #####
#######################################################

glueContext.sql(f"""
INSERT INTO {CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}_nopartitions
SELECT c_customer_sk, c_customer_id, c_first_name, c_last_name, c_birth_country, c_email_address
FROM another_table
""")

错误日志 #

2025-01-30 20:53:44,898 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(77)): Error from Python:Traceback (most recent call last):
  File "/tmp/main.py", line 85, in <module>
    spark.sql(f"""
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1034, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o104.sql.
: org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore
    at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:84)

已尝试的无效操作 #

• --enable-glue-datacatalog = true 无效果
• .enableHiveSupport() 无效果
• 将catalog设置为SparkSessionCatalog
• 尝试在所有场景使用glueSession.sql和spark.sql
• 查阅多个StackOverflow帖子（但均为Hive相关场景）
• 阅读大量文档

问题核心是Spark默认spark_catalog的配置冲突，以及Glue Job运行时的Iceberg配置优先级问题，以下是具体修复步骤：

1. 修正Spark Catalog配置 #

你当前将spark.sql.catalog.spark_catalog设为org.apache.iceberg.spark.SparkSessionCatalog，但这个类默认绑定Hive Metastore作为底层存储，导致Iceberg尝试连接Hive。需要调整默认Catalog的配置：

方案A：将自定义glue_catalog设为默认 #

在SparkConf中添加一行，让所有未指定Catalog的SQL操作默认使用Glue Catalog：

.set("spark.sql.defaultCatalog", CATALOG_NAME)

方案B：修正spark_catalog的Glue配置 #

如果要保留spark_catalog作为默认，需要补充完整的Glue Catalog参数：

.set("spark.sql.catalog.spark_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
.set("spark.sql.catalog.spark_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.set("spark.sql.catalog.spark_catalog.warehouse", "s3://<your-warehouse-dir>")

2. 配置Glue Job运行参数 #

在Glue Job的Job parameters中添加以下参数，确保运行时强制使用Glue Catalog：

• --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
• --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

注意：不要添加任何Hive相关参数，比如--enable-hive-support

3. 明确SQL中的表引用 #

INSERT语句中查询的another_table如果是Glue Catalog中的表，必须明确指定Catalog路径，比如glue_catalog.default.another_table，否则Spark会用默认Catalog查找，触发错误的Hive连接逻辑。

4. 修复后的完整代码示例 #

from pyspark.sql import SparkSession
from pyspark import SparkConf
from awsglue.context import GlueContext

DB_NAME='default'
CATALOG_NAME="glue_catalog"
TABLE_NAME = "table1"

conf = (SparkConf().setAppName("Spark Test") 
    .set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .set(f"spark.sql.catalog.{CATALOG_NAME}", "org.apache.iceberg.spark.SparkCatalog")
    .set(f"spark.sql.catalog.{CATALOG_NAME}.warehouse", "s3://<your-warehouse-dir>")
    .set(f"spark.sql.catalog.{CATALOG_NAME}.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") 
    .set(f"spark.sql.catalog.{CATALOG_NAME}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    # 设置默认Catalog为自定义的glue_catalog
    .set("spark.sql.defaultCatalog", CATALOG_NAME)
    )

spark = (SparkSession 
    .builder 
    .appName("Python Spark Iceberg example") 
    .config(conf=conf) 
    .getOrCreate() 
)
        
glueContext = GlueContext(spark.sparkContext)

# 创建表（无需重复指定Catalog，默认使用glue_catalog）
glueContext.sql(f"""
    CREATE TABLE IF NOT EXISTS {DB_NAME}.{TABLE_NAME}_nopartitions (
        c_customer_sk             int,
        c_customer_id             string,
        c_first_name              string,
        c_last_name               string,
        c_birth_country           string,
        c_email_address           string)
    USING iceberg
    OPTIONS ('format-version'='2')
""")

# 插入数据，明确指定源表的Catalog路径
glueContext.sql(f"""
INSERT INTO {DB_NAME}.{TABLE_NAME}_nopartitions
SELECT c_customer_sk, c_customer_id, c_first_name, c_last_name, c_birth_country, c_email_address
FROM {CATALOG_NAME}.{DB_NAME}.another_table
""")

内容的提问来源于stack exchange，提问作者john k