Spark on K8s Operator环境下PySpark写入Hudi到ABFS时出现HoodieUpsertException问题求助

问题背景 #

最近我在使用Spark on K8s Operator提交Spark应用，尝试将处理后的数据通过Hudi写入Azure Data Lake Storage Gen2（ABFS路径：abfs://container@storageaccount.dfs.core.windows.net/folder/host）时遇到了任务失败的问题。应用依赖的Jar包版本如下：

• com.microsoft.azure:azure-storage:8.6.6
• org.apache.hadoop:hadoop-azure:3.3.1
• org.apache.hadoop:hadoop-common:3.3.1
• org.apache.hudi:hudi-spark3-bundle_2.12:0.10.0

错误详情 #

应用运行时反复出现任务失败，最终抛出的核心错误信息如下：

22/06/08 20:45:15 ERROR TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job
Hudi写入失败：host
调用o146.save时发生错误：
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20220608204512566
...
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 8) (10.42.5.12 executor 1): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD

解决办法 #

经过排查，发现问题根源在于Spark on K8s Operator环境与Driver/Executor的依赖版本不一致。我通过以下操作解决了问题：

• 不再依赖spark-submit的依赖配置（比如--jars或spark.jars参数）动态加载Hadoop、Azure存储和Hudi的Jar包
• 改为在构建Spark Driver和Executor的镜像时，将所有所需的依赖Jar包直接内置到镜像中，确保Operator运行环境与应用运行时的依赖版本完全统一

调整后，ClassCastException不再出现，Hudi数据成功写入ABFS存储路径。

内容的提问来源于stack exchange，提问作者devbot15