使用Apache Iceberg时遭遇Hive依赖缺失问题求助

环境配置 #

SBT依赖 #

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
libraryDependencies += "org.apache.iceberg" % "iceberg-spark-runtime-3.2_2.12" % "0.13.2"

Spark会话配置及写入代码 #

val builder = SparkSession
      .builder()
      .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
      .config("spark.sql.catalog.spark_catalog","org.apache.iceberg.spark.SparkSessionCatalog")
      .config("spark.sql.catalog.spark_catalog.type","hive")
      .config("spark.sql.catalog.local","org.apache.iceberg.spark.SparkCatalog")
      .config("spark.sql.catalog.local.type","hadoop")
      .config("spark.sql.catalog.local.warehouse","/Users/tom/Documents/hive/warehouse")
      .getOrCreate()

import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.functions._

// 创建DataFrame
val data = Seq(
("100", "2015-01-01", "2015-01-01T13:51:39.340396Z"),
("101", "2015-01-01", "2015-01-01T12:14:58.597216Z"),
("102", "2015-01-01", "2015-01-01T13:51:40.417052Z"),
("103", "2015-01-01", "2015-01-01T13:51:40.519832Z")
).toDF("id", "creation_date", "last_update_time")

data.write
.format("iceberg")
.save("/Users/tom/Documents/data") 

报错信息 #

代码运行到save步骤时触发以下错误：

Cannot initialize Catalog implementation org.apache.iceberg.hive.HiveCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
    Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError: org/apache/thrift/TException]
java.lang.IllegalArgumentException: Cannot initialize Catalog implementation org.apache.iceberg.hive.HiveCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
    Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError: org/apache/thrift/TException]
    at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:182)
    at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:234)
    at org.apache.iceberg.spark.SparkCatalog.buildIcebergCatalog(SparkCatalog.java:119)
    at org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:411)
    at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:60)
    at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$catalog$1(CatalogManager.scala:52)
    at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
    at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:52)
    at org.apache.iceberg.spark.source.IcebergSource.catalogAndIdentifier(IcebergSource.java:129)
    at org.apache.iceberg.spark.source.IcebergSource.extractIdentifier(IcebergSource.java:159)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:290)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
    ...（省略后续栈跟踪）
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
    Missing org.apache.iceberg.hive.HiveCatalog [java.lang.NoClassDefFoundError: org/apache/thrift/TException]

问题分析及解决方案 #

核心原因 #

1. 配置中指定默认spark_catalog.type为hive，但当前依赖缺少Hive Catalog必需的Thrift及Hive客户端依赖
2. iceberg-spark-runtime仅包含Spark集成核心依赖，不包含Hive Catalog的完整依赖；单独引入iceberg-hive-runtime可能因版本不匹配或依赖传递不全导致问题

解决方案 #

方案1：改用已配置的Hadoop Catalog写入

无需Hive依赖，直接使用local Hadoop Catalog写入，修改代码如下：

// 写入到local catalog的指定表
data.write
.format("iceberg")
.mode(SaveMode.Append)
.save("local.db.test_table")

或写入指定路径时显式指定Catalog：

data.write
.format("iceberg")
.option("catalog", "local")
.save("/Users/tom/Documents/data")

方案2：补充Hive Catalog完整依赖

若必须使用Hive Catalog，更新SBT依赖，引入兼容版本的Hive runtime：

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
libraryDependencies += "org.apache.iceberg" % "iceberg-spark-runtime-3.2_2.12" % "0.13.2",
libraryDependencies += "org.apache.iceberg" % "iceberg-hive-runtime-3.2_2.12" % "0.13.2"

同时将Hive配置文件（如hive-site.xml）放置到Spark的conf目录，确保Hive客户端可正常初始化。

方案3：移除Hive Catalog配置

若无需Hive作为默认Catalog，修改Spark会话配置，删除Hive相关项：

val builder = SparkSession
      .builder()
      .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
      .config("spark.sql.catalog.local","org.apache.iceberg.spark.SparkCatalog")
      .config("spark.sql.catalog.local.type","hadoop")
      .config("spark.sql.catalog.local.warehouse","/Users/tom/Documents/hive/warehouse")
      .getOrCreate()

内容的提问来源于stack exchange，提问作者Leroy Mikenzi