Spark作业自定义日志无法写入EventLogs，History Server无显示求助

我尝试将Spark作业的日志输出到Spark EventLogs，以便在History Server中查看，采用了两种方案：

• 实现继承Serializable的自定义Logger
• 在对象中继承org.apache.spark.internal.Logging特质

相关代码 #

MyCustomLogger对象 #

package com.vikas

import org.apache.logging.log4j.Logger
import org.apache.logging.log4j.LogManager
import org.apache.logging.log4j.core.appender.ConsoleAppender

object MyCustomLogger extends Serializable {  
  @transient lazy val log = LogManager.getRootLogger()
}

WordCount对象 #

package com.vikas

import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import org.apache.logging.log4j.Logger
import org.apache.logging.log4j.LogManager
import org.apache.logging.log4j.core.config.LoggerConfig
import org.apache.logging.log4j.core.config.Configurator
import org.apache.spark.internal.Logging

object WordCount extends Logging {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().appName("WordCountExample").getOrCreate()
    val filePath = args(0).trim

    WordCount.log.warn("Vikas: reading files")
    val lines = spark.sparkContext.textFile(filePath)

    MyCustomLogger.log.warn("transforming lines")
    val wordCounts = transform(lines)

    import java.time.LocalDateTime
    import java.time.format.DateTimeFormatter
    val currdate: String = DateTimeFormatter.ofPattern("yyyyMMdd_HHmm").format(LocalDateTime.now)
    val outputDir = args(1).trim+currdate

    MyCustomLogger.log.warn("writing output")
    wordCounts.coalesce(1).saveAsTextFile(outputDir)

    MyCustomLogger.log.warn("finished writing output")
  }

  def transform(rdd: RDD[String]): RDD[(String, Int)] = {
    rdd
    .flatMap(_.split("\\s"))    // Split on any white character
    .map(_.replaceAll(
      "[,.!?:;]", "")           // Remove punctuation and transfer to lowercase
      .trim
      .toLowerCase)
    .filter(!_.isEmpty)         // Filter out any non-words
    .map((_, 1))                // Initialize word count pairs
    .reduceByKey(_ + _)         // Finally, count words
    .sortByKey()                // and sort the word counts in a lexical order
  }
}

log4j2.properties文件 #

rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console

appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

问题现象 #

运行代码后，History Server的作业日志中看不到自定义日志输出，以下是History Server日志内容：

24/09/10 06:19:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/09/10 06:19:07 WARN DependencyUtils: Skip remote jar s3://vikasemrbucket/jars/SampleSparkProject-1.0.jar.
24/09/10 06:19:07 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at ip-172-31-20-226.ap-southeast-2.compute.internal/172.31.20.226:8032
24/09/10 06:19:08 INFO Configuration: resource-types.xml not found
24/09/10 06:19:08 INFO ResourceUtils: Unable to find 'resource-types.xml'.
24/09/10 06:19:08 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (12288 MB per container)
24/09/10 06:19:08 INFO Client: Will allocate AM container, with 2432 MB memory including 384 MB overhead
24/09/10 06:19:08 INFO Client: Setting up container launch context for our AM
24/09/10 06:19:08 INFO Client: Setting up the launch environment for our AM container
24/09/10 06:19:08 INFO Client: Preparing resources for our AM container
24/09/10 06:19:08 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
24/09/10 06:19:11 INFO Client: Uploading resource file:/mnt/tmp/spark-60dda2ce-2cc0-41b4-9800-9a8e679e3833/__spark_libs__429154216109170134.zip -> hdfs://ip-172-31-20-226.ap-southeast-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1725949012477_0002/__spark_libs__429154216109170134.zip
24/09/10 06:19:13 INFO ClientConfigurationFactory: Set initial getObject socket timeout to 2000 ms.
24/09/10 06:19:13 INFO Client: Uploading resource s3://vikasemrbucket/jars/SampleSparkProject-1.0.jar -> hdfs://ip-172-31-20-226.ap-southeast-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1725949012477_0002/SampleSparkProject-1.0.jar
24/09/10 06:19:14 INFO S3NativeFileSystem: Opening 's3://vikasemrbucket/jars/SampleSparkProject-1.0.jar' for reading
24/09/10 06:19:15 INFO Client: Uploading resource file:/etc/hudi/conf.dist/hudi-defaults.conf -> hdfs://ip-172-31-20-226.ap-southeast-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1725949012477_0002/hudi-defaults.conf
24/09/10 06:19:16 INFO Client: Uploading resource file:/mnt/tmp/spark-60dda2ce-2cc0-41b4-9800-9a8e679e3833/__spark_conf__6454346447521834005.zip -> hdfs://ip-172-31-20-226.ap-southeast-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1725949012477_0002/__spark_conf__.zip
24/09/10 06:19:16 INFO SecurityManager: Changing view acls to: hadoop
24/09/10 06:19:16 INFO SecurityManager: Changing modify acls to: hadoop
24/09/10 06:19:16 INFO SecurityManager: Changing view acls groups to: 
24/09/10 06:19:16 INFO SecurityManager: Changing modify acls groups to: 
24/09/10 06:19:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
24/09/10 06:19:16 INFO Client: Submitting application application_1725949012477_0002 to ResourceManager
24/09/10 06:19:16 INFO YarnClientImpl: Submitted application application_1725949012477_0002
24/09/10 06:19:17 INFO Client: Application report for application_1725949012477_0002 (state: ACCEPTED)
24/09/10 06:19:17 INFO Client: 
     client token: N/A
     diagnostics: AM container is launched, waiting for AM container to Register with RM
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1725949156784
     final status: UNDEFINED
     tracking URL: http://ip-172-31-20-226.ap-southeast-2.compute.internal:20888/proxy/application_1725949012477_0002/
     user: hadoop
24/09/10 06:19:18 INFO Client: Application report for application_1725949012477_0002 (state: ACCEPTED)
24/09/10 06:19:19 INFO Client: Application report for application_1725949012477_0002 (state: ACCEPTED)
24/09/10 06:19:20 INFO Client: Application report for application_1725949012477_0002 (state: ACCEPTED)
24/09/10 06:19:21 INFO Client: Application report for application_1725949012477_0002 (state: ACCEPTED)
24/09/10 06:19:22 INFO Client: Application report for application_1725949012477_0002 (state: ACCEPTED)
24/09/10 06:19:23 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:23 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: ip-172-31-29-193.ap-southeast-2.compute.internal
     ApplicationMaster RPC port: 34877
     queue: default
     start time: 1725949156784
     final status: UNDEFINED
     tracking URL: http://ip-172-31-20-226.ap-southeast-2.compute.internal:20888/proxy/application_1725949012477_0002/
     user: hadoop
24/09/10 06:19:24 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:25 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:26 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:27 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:28 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:29 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:30 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:31 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:32 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:33 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:34 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:35 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:36 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:37 INFO Client: Application report for application_1725949012477_0002 (state: RUNNING)
24/09/10 06:19:38 INFO Client: Application report for application_1725949012477_0002 (state: FINISHED)
24/09/10 06:19:38 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: ip-172-31-29-193.ap-southeast-2.compute.internal
     ApplicationMaster RPC port: 34877
     queue: default
     start time: 1725949156784
     final status: SUCCEEDED
     tracking URL: http://ip-172-31-20-226.ap-southeast-2.compute.internal:20888/proxy/application_1725949012477_0002/
     user: hadoop
24/09/10 06:19:38 INFO ShutdownHookManager: Shutdown hook called
24/09/10 06:19:38 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-fd81d76f-bb66-490e-b4be-83149cb4ea4a
24/09/10 06:19:38 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-60dda2ce-2cc0-41b4-9800-9a8e679e3833
Command exiting with ret '0'

核心问题分析 #

Spark History Server展示的EventLogs默认只捕获两类日志：一是通过org.apache.spark.internal.Logging接口输出的Spark体系日志，二是YARN容器的标准输出/错误流。你的自定义日志未被捕获，根源在于：

1. 自定义Logger独立于Spark日志体系：用LogManager.getRootLogger()获取的是独立Log4j2实例，输出仅到控制台，未接入Spark的EventLog收集机制。
2. Logging特质使用错误：直接通过WordCount.log调用日志方法，未正确绑定Spark日志上下文，导致日志未写入EventLogs。
3. EventLog收集未确认开启：若Spark未配置EventLog相关参数，即使日志输出正确，也无法在History Server中查看。

具体解决步骤 #

方案一：正确使用org.apache.spark.internal.Logging特质 #

修改WordCount对象的日志调用方式，直接使用特质提供的log方法（无需对象前缀）：

object WordCount extends Logging {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().appName("WordCountExample").getOr