使用Flink通过Hive Catalog写入Paimon表遇ClassNotFoundException求助

异常堆栈信息 #

java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:4051)
    at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:4019)
    at org.apache.paimon.hive.HiveCatalog.createHiveConf(HiveCatalog.java:697)
    at org.apache.paimon.hive.HiveCatalog.createHiveConf(HiveCatalog.java:756)
    at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:710)
    at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:50)
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76)
    at org.apache.paimon.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:69)
    at org.apache.paimon.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:59)
    at org.apache.paimon.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:32)
    at org.apache.flink.table.factories.FactoryUtil.createCatalog(FactoryUtil.java:488)

问题原因 #

Hive依赖存放在Flink JVM类路径中，而Hadoop依赖存放在pipeline.classpaths路径下：

• org.apache.hadoop.hive.conf.HiveConf由AppClassLoader加载
• org.apache.hadoop.mapred.JobConf需要由FlinkUserCodeClassLoader加载

已尝试的无效方案 #

已知FlinkUserCodeClassLoader的父类加载器是AppClassLoader，因此在初始化HiveConf前添加了以下代码，期望让HiveConf和JobConf都能被正确加载，但仍出现相同异常：

Thread.currentThread().setContextClassLoader(flinkUserCodeClassLoader);

约束条件 #

• Hadoop和Flink依赖的存储位置无法轻易更改
• 必须保证生产环境中Flink作业的兼容性

求助 #

恳请熟悉类加载器原理的人士指点如何解决该问题？

内容的提问来源于stack exchange，提问作者zbz zbz