PySpark+Oozie运行报错：依赖spark-solr jar无法实例化Hive元存储客户端

先明确下你的场景：PySpark代码依赖spark-solr shaded jar，Yarn命令行跑完全正常，但放到Oozie里就报错，核心问题是Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient。这种"本地/命令行正常、调度工具异常"的情况，大多和类加载顺序或环境配置传递有关，下面给你几个针对性的解决方案：

错误栈回顾 #

先把你遇到的错误贴出来方便参考：

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext. : java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) 
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:209) 
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238) 
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:225) 
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:215) 
at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:480) 
at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:479) 
at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40) 
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330) 
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90) 
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) 
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) 
at py4j.Gateway.invoke(Gateway.java:214) 
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) 
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) 
at py4j.GatewayConnection.run(GatewayConnection.java:209) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) 
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) 
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) 
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) 
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) 
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) 
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) 
... 21 more 
Caused by: java.lang.reflect.InvocationTargetException 
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) 
... 27 more 
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught. 
NestedThrowables: 
java.lang.reflect.InvocationTargetException 
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1193) 
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) 
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) 
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365) 
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394) 
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291) 
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258) 
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) 
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) 
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:57) 
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66) 
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593) 
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571) 
at org.apache.hadoop.hive.metast

核心原因分析 #

问题出在类加载冲突：spark-solr的shaded jar里可能包含了和集群Hive metastore依赖冲突的类（比如JDO、Guava或者Hive自身的类）。命令行环境下，Spark会优先加载集群自带的Hive依赖，但Oozie调度时，--jars指定的用户jar可能会覆盖系统类，导致Hive metastore无法正确实例化。

解决方案 #

1. 强制Hive相关类使用系统类加载器 #

这是最快速的解决方案，不需要修改jar包。在Oozie的spark-opts中添加Spark配置，指定Hive metastore相关的包由共享类加载器加载，避免用户jar里的类干扰：

--conf spark.sql.hive.metastore.sharedPrefixes=javax.jdo,org.apache.hadoop.hive,org.apache.hadoop.hive.metastore,com.mysql.cj.jdbc,org.postgresql

（最后两个数据库驱动包根据你的Hive metastore使用的数据库调整，比如MySQL就加com.mysql.cj.jdbc，PostgreSQL就加org.postgresql）

把这个配置加到你Oozie workflow的spark-opts里，示例：

<spark-opts>--jars spark-solr-shaded-4.10.0.jar --conf spark.sql.hive.metastore.sharedPrefixes=javax.jdo,org.apache.hadoop.hive,org.apache.hadoop.hive.metastore,com.mysql.cj.jdbc</spark-opts>

2. 重新打包排除冲突依赖（如果有条件） #

如果你能重新构建spark-solr，可以在打包时排除和Hive冲突的依赖，这样shaded jar里就不会包含这些类，自然不会和集群的Hive依赖冲突。在Maven的pom.xml中添加排除规则：

<dependency>
    <groupId>com.lucidworks.spark</groupId>
    <artifactId>spark-solr</artifactId>
    <version>你的版本号</version>
    <exclusions>
        <!-- 排除Hive相关依赖 -->
        <exclusion>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-metastore</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
        </exclusion>
        <!-- 排除JDO依赖 -->
        <exclusion>
            <groupId>javax.jdo</groupId>
            <artifactId>jdo-api</artifactId>
        </exclusion>
        <!-- 排除Guava（如果版本冲突的话） -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
        </exclusion>
    </exclusions>
</dependency>

然后用mvn package shade:shade重新打包，得到的jar就不会包含这些冲突类了。

3. 确保Oozie传递Hive配置文件 #

有时候Oozie任务没有正确获取hive-site.xml，导致Hive metastore的配置缺失，也会引发这个错误。在Oozie的workflow.xml中，给Spark节点添加<file>标签，引用集群的hive-site.xml：

<action name="pyspark-solr-job">
    <spark xmlns="uri:oozie:spark-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <master>yarn</master>
        <name>PySpark Solr Job</name>
        <jar>your-spark-script.py</jar>
        <spark-opts>--jars spark-solr-shaded.jar --conf spark.sql.hive.metastore.sharedPrefixes=javax.jdo,org.apache.hadoop.hive,org.apache.hadoop.hive.metastore</spark-opts>
        <!-- 添加hive-site.xml文件 -->
        <file>/etc/hive/conf/hive-site.xml#hive-site.xml</file>
    </spark>
    <ok to="end"/>
    <error to="fail"/>
</action>

这样Spark任务就能读取到正确的Hive metastore地址、数据库配置等信息。

4. 切换到SparkSession API（推荐） #

如果你的代码还在使用旧的HiveContext，建议切换到Spark 2.x+推荐的SparkSession，它对Hive metastore的兼容性更好，也能避免一些旧API的类加载问题：

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Solr Data Processing") \
    .enableHiveSupport() \
    .getOrCreate()

# 之后用spark代替HiveContext即可，比如：
# df = spark.read.table("your_hive_table")

内容的提问来源于stack exchange，提问作者Chirag