如何用log4j2 RegexFilter过滤Spark的Truncated日志警告？

我尝试用Log4j2的RegexFilter过滤Spark的警告日志：Truncated the string representation of a plan since it was too long。为了避免应用日志出现查询计划输出，我设置了spark.sql.maxPlanStringLength=0，结果触发了这个警告。

我编写了触发该警告的Spark Scala应用：

package sparklog4j2

import org.apache.spark.sql.SparkSession
import org.apache.logging.log4j.core.LoggerContext
import org.apache.logging.log4j.core.config.{LoggerConfig}
import org.apache.logging.log4j.{Logger, LogManager, Level}

object Demo {
  def main(args: Array[String]): Unit = {
    val ctx: LoggerContext = LogManager.getContext().asInstanceOf[LoggerContext]
    val conf = ctx.getConfiguration()
    println(s"CONFIG NAME: ${conf.getName}")
    val spark = SparkSession.builder().appName("log4j2 demo").getOrCreate()
    import spark.implicits._
    spark.createDataset[String](Seq("foo","bar")).show
  }
}

通过sbt assembly构建fat jar，依赖配置如下：

scalaVersion := "2.12.15"

version := "1.0.0"

libraryDependencies ++= Seq(
  "org.apache.logging.log4j" % "log4j-api" % "2.13.2",
  "org.apache.logging.log4j" % "log4j-core" % "2.13.2",
  "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.13.2",
  "org.apache.logging.log4j" % "log4j-1.2-api" % "2.13.2" % "provided",
  "org.apache.spark" %% "spark-core" % "3.2.1" % "provided",
  "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
)

我的log4j2.json配置了全局RegexFilter：

{
  "configuration": {
    "name": "sparklog4j2-demo",
    "RegexFilter": {
      "regex": ".*Truncated.*",
      "onMatch": "DENY",
      "onMismatch": "NEUTRAL"
    },
    "loggers": {
      "logger": [
        {
          "name": "org.apache.spark.*",
          "level": "error",
          "includeLocation": true
        }
      ],
      "root": {
        "level": "error",
        "includeLocation": true
      }
    }
  }
}

运行命令：

spark-submit \
--verbose \
--class sparklog4j2.Demo \
--jars ./jars/log4j-1.2-api-2.13.2.jar \
--driver-java-options "-Dlog4j.configurationFile=files/log4j2.json -Dlog4j2.debug=true -DLog4jDefaultStatusLevel=trace" \
--conf "spark.sql.maxPlanStringLength=0" \
--files ./files/log4j2.json \
target/scala-2.12/log4j-spark-assembly-1.0.0.jar

运行时出现链接错误：

INFO StatusLogger Plugin [org.apache.hadoop.hive.ql.log.HiveEventCounter] could not be loaded due to linkage error.
java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/appender/AbstractAppender

虽然应用打印了CONFIG NAME: sparklog4j2-demo证明加载了自定义配置，但Spark还是输出：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
...
WARN StringUtils: Truncated the string representation of a plan since it was too long.

过滤器没生效，Spark似乎没使用我的log4j配置，该怎么解决？

1. 修正Log4j2配置的过滤器位置 #

Log4j2的全局RegexFilter不能直接放在configuration节点下，必须绑定到Appender上才能生效。修改后的log4j2.json需要添加Appender定义，并将过滤器附加到Appender中：

{
  "configuration": {
    "name": "sparklog4j2-demo",
    "appenders": {
      "Console": {
        "name": "ConsoleAppender",
        "target": "SYSTEM_OUT",
        "PatternLayout": {
          "pattern": "%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n"
        },
        "RegexFilter": {
          "regex": ".*Truncated the string representation of a plan since it was too long.*",
          "onMatch": "DENY",
          "onMismatch": "ACCEPT"
        }
      }
    },
    "loggers": {
      "logger": [
        {
          "name": "org.apache.spark",
          "level": "error",
          "additivity": false,
          "AppenderRef": {
            "ref": "ConsoleAppender"
          }
        },
        {
          "name": "org.apache.spark.sql.util.StringUtils",
          "level": "error",
          "additivity": false,
          "AppenderRef": {
            "ref": "ConsoleAppender"
          }
        }
      ],
      "root": {
        "level": "error",
        "AppenderRef": {
          "ref": "ConsoleAppender"
        }
      }
    }
  }
}

• 单独针对org.apache.spark.sql.util.StringUtils（输出该警告的Logger）设置级别为error，比RegexFilter更高效，可直接屏蔽该警告。
• 将onMismatch改为ACCEPT，确保非匹配日志正常输出。

2. 强制Spark使用自定义Log4j2配置 #

Spark启动时会优先加载内置的log4j-defaults.properties，需通过以下方式确保自定义配置覆盖默认值：

调整Spark提交命令 #

修改spark-submit命令，将Log4j2相关参数前置，并添加Spark配置强制指定日志加载规则：

spark-submit \
--verbose \
--class sparklog4j2.Demo \
--driver-java-options "-Dlog4j.configurationFile=./files/log4j2.json -Dlog4j2.debug=true -DLog4jDefaultStatusLevel=trace -Dspark.driver.extraClassPath=./jars/log4j-core-2.13.2.jar:./jars/log4j-api-2.13.2.jar:./jars/log4j-slf4j-impl-2.13.2.jar" \
--conf "spark.sql.maxPlanStringLength=0" \
--conf "spark.driver.userClassPathFirst=true" \
--conf "spark.executor.userClassPathFirst=true" \
--files ./files/log4j2.json \
--jars ./jars/log4j-core-2.13.2.jar,./jars/log4j-api-2.13.2.jar,./jars/log4j-slf4j-impl-2.13.2.jar \
target/scala-2.12/log4j-spark-assembly-1.0.0.jar

• 添加spark.driver.userClassPathFirst=true和spark.executor.userClassPathFirst=true：强制Spark优先加载用户提供的依赖，避免使用Spark内置的旧版本Log4j。
• 将所有Log4j2相关jar包添加到--jars和driver.extraClassPath中，确保Driver和Executor都能加载到完整的Log4j2依赖。

3. 修复依赖冲突问题 #

Spark 3.2.1内置的日志组件可能与你的Log4j2版本存在冲突，调整sbt依赖：

libraryDependencies ++= Seq(
  "org.apache.logging.log4j" % "log4j-api" % "2.13.2" % "provided",
  "org.apache.logging.log4j" % "log4j-core" % "2.13.2" % "provided",
  "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.13.2" % "provided",
  "org.apache.logging.log4j" % "log4j-1.2-api" % "2.13.2" % "provided",
  "org.apache.spark" %% "spark-core" % "3.2.1" % "provided",
  "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
)

• 将所有Log4j2依赖标记为provided，避免fat jar打包这些依赖，而是通过--jars参数在提交时指定，减少冲突概率。

4. 验证配置是否生效 #

运行应用时，检查日志输出：

• 若不再看到Using Spark's default log4j profile的提示，说明自定义配置已成功加载。
• WARN StringUtils: Truncated...的日志应该被过滤掉。

内容的提问来源于stack exchange，提问作者David Farrell