Scala测试创建本地Spark Session时出现log4j Appender创建失败错误

在使用FunSuite进行Scala测试、创建本地Spark Session时，抛出如下错误：

log4j:ERROR 无法创建Appender。报告的错误如下。
java.lang.ClassNotFoundException: com.microsoft.applicationinsights.log4j.v1_2.ApplicationInsightsAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:247)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191)
at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523)
at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:492)
at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1006)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:872)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
at org.apache.spark.network.util.JavaUtils.(JavaUtils.java:41)
at org.apache.spark.internal.config.ConfigHelpers$.byteFromString(ConfigBuilder.scala:67)
at org.apache.spark.internal.config.ConfigBuilder.$anonfun$bytesConf$1(ConfigBuilder.scala:259)
at org.apache.spark.internal.config.ConfigBuilder.$anonfun$bytesConf$1$adapted(ConfigBuilder.scala:259)
at org.apache.spark.internal.config.TypedConfigBuilder.$anonfun$transform$1(ConfigBuilder.scala:101)
at org.apache.spark.internal.config.TypedConfigBuilder.createWithDefault(ConfigBuilder.scala:144)
at org.apache.spark.internal.config.package$.(package.scala:345)
at org.apache.spark.internal.config.package$.(package.scala)
at org.apache.spark.SparkConf$.(SparkConf.scala:654)
at org.apache.spark.SparkConf$.(SparkConf.scala)
at org.apache.spark.SparkConf.set(SparkConf.scala:94)
at org.apache.spark.SparkConf.set(SparkConf.scala:83)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:916)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:916)
at com.random.SampleSpec.(SampleSpec.scala:9)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:66)
at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.scalatest.tools.DiscoverySuite.(DiscoverySuite.scala:37)
at org.scalatest.tools.Runner$.genDiscoSuites$1(Runner.scala:1128)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1224)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:993)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:971)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1480)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:971)
at org.scalatest.tools.Runner$.main(Runner.scala:775)
at org.scalatest.tools.Runner.main(Runner.scala)

测试代码片段 #

import org.apache.spark.sql.{DataFrame, SparkSession}
import org.scalatest.BeforeAndAfter
import org.scalatest.funsuite.AnyFunSuite

class SampleSpec extends AnyFunSuite with BeforeAndAfter {
  val spark: SparkSession = SparkSession.builder().master("local[*]").appName("SampleTest").getOrCreate()
  spark.conf.set("spark.sql.crossJoin.enabled", "true")
  spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")
  
  before {
    spark.sparkContext.setLogLevel("ERROR")
    logger.info("Before")
  }
}

pom.xml相关配置 #

<!-- Test -->
<!-- https://mvnrepository.com/artifact/org.scalatest/scalatest -->
<dependency>
    <groupId>org.scalatest</groupId>
    <artifactId>scalatest_${scala.version.major}</artifactId>
    <version>3.3.0-SNAP3</version>
    <scope>test</scope>
</dependency>

<!-- https://mvnrepository.com/artifact/org.scalacheck/scalacheck -->
<dependency>
    <groupId>org.scalacheck</groupId>
    <artifactId>scalacheck_${scala.version.major}</artifactId>
    <version>1.15.4</version>
    <scope>test</scope>
</dependency>

<plugin>
    <groupId>org.scalatest</groupId>
    <artifactId>scalatest-maven-plugin</artifactId>
    <version>2.0.0</version>
    <configuration>
        <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
        <junitxml>.</junitxml>
        <filereports>WDF TestSuite.txt</filereports>
    </configuration>
    <executions>
        <execution>
            <id>test</id>
            <goals>
                <goal>test</goal>
            </goals>
        </execution>
    </executions>
</plugin>

环境与已尝试操作 #

• 运行环境：IntelliJ 2021.2.3
• Spark版本：3.1.3（本地安装与pom配置一致）
• Scala版本：2.12
• 已尝试：复制旧模块配置、Invalidate and Restart、重新构建模块、添加Scala支持框架，问题仍未解决；相同代码在旧模块可正常运行。

解决方案 #

• 添加缺失依赖：在pom.xml的test依赖中加入Application Insights的log4j适配器，版本选择与Spark兼容的版本：

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-logging-log4j1_2</artifactId>
    <version>2.6.4</version>
    <scope>test</scope>
</dependency>

• 自定义log4j配置：在src/test/resources目录下创建log4j.properties，移除对ApplicationInsightsAppender的引用，使用基础控制台输出：

log4j.rootLogger=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

• 排查依赖冲突：执行mvn dependency:tree命令对比新旧模块的依赖树，找出缺失或冲突的依赖项，统一依赖版本。

内容的提问来源于stack exchange，提问作者Sharad R. Telkar