Spark/Java项目单元测试出现MapR相关类找不到的问题求助
Spark/Java项目单元测试出现MapR相关类找不到的问题求助
大家好,我现在碰到个头疼的问题,想请各位帮忙支支招:
我正在维护一个Java/Spark项目,目前打算给它补充单元测试。项目里用到的核心组件版本如下:
<spark.version>3.5.2.0</spark.version> <hive.version>3.1.3.650</hive.version> <hbase.version>1.4.14.700</hbase.version> <hadoop.version>3.3.5.500</hadoop.version>
为了适配环境,我已经做了这些配置:
- 配置了环境变量
MAPR_HOME="C:\Users\USER\opt\mapr" - 在
C:\Users\<USER>\opt\mapr\conf\mapr-clusters.conf路径下放置了一个空的mapr-clusters.conf文件
但当我执行单元测试时,接连抛出了三个错误:
第一个错误:
java.lang.ClassNotFoundException: com.mapr.fs.ShimLoader at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:61) at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:804) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:251) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:666) at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416) at org.apache.spark.SparkContext.<init>(SparkContext.scala:331) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2908) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099) at scala.Option.getOrElse(Option.scala:189)
第二个错误:
java.lang.ClassNotFoundException: com.mapr.baseutils.JVMProperties at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:78) at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:804) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:251) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:666) at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
第三个错误:
java.lang.NoClassDefFoundError: com/mapr/web/security/SslConfig
有没有朋友遇到过类似的问题?或者能给我一些排查方向的建议?非常感谢!
备注:内容来源于stack exchange,提问作者Marcel kobain
相关产品推荐
相关产品推荐

