Spark on Kubernetes提交后初始无级别日志设为INFO级别求助

我在Kubernetes上运行Spark应用，日志已正确发送至Datadog，但spark-submit后的前5行日志未附带级别属性，被Datadog识别为Error类型：

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.2.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

其余日志通过Log4j发送并附带级别属性，示例如下：

{
  "endOfBatch": false,
  "instant": {
    "epochMillisecond": 1686055630113.133,
    "epochSecond": 1686055630,
    "nanoOfSecond": 113133000
  },
  "level": "INFO",
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "loggerName": "com.package.Main",
  "thread": "main",
  "threadId": 1,
  "threadPriority": 5
}

请问如何将这些初始日志标记为INFO级别？

这些初始日志是JVM启动时的非法反射访问警告，不属于Spark的Log4j日志输出，因此没有携带级别属性，Datadog会默认将无级别标记的日志识别为Error。以下是三种可行的解决方法：

方法1：从源头消除警告 #

直接通过JVM参数禁用这类非法反射警告，彻底避免日志产生：

spark-submit \
  --conf "spark.driver.extraJavaOptions=--add-opens java.base/java.nio=ALL-UNNAMED --illegal-access=deny" \
  --conf "spark.executor.extraJavaOptions=--add-opens java.base/java.nio=ALL-UNNAMED --illegal-access=deny" \
  # 补充你的其他spark-submit参数

--add-opens参数允许Spark的unsafe模块访问JDK内部API，--illegal-access=deny会关闭后续的非法反射警告输出。

方法2：在Datadog端修正日志级别 #

通过Datadog的日志流水线规则，匹配这些特定警告并强制设置为INFO级别：

1. 进入Datadog控制台的日志>流水线页面
2. 添加新的「修改日志级别」处理器
3. 设置匹配规则：message matches "WARNING: An illegal reflective access operation has occurred|WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform.*|WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform|WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations|WARNING: All illegal access operations will be denied in a future release"
4. 将目标级别设置为INFO
5. 保存规则并应用到对应的日志管道

方法3：将JVM输出接入Spark的Log4j #

修改Spark的Log4j配置，把JVM的stdout/stderr输出重定向到Log4j，让这些日志自动带上INFO级别：

1. 编辑Spark的Log4j配置文件（如log4j2.properties），添加以下配置：

# 捕获JVM stdout并标记为INFO级别
appender.jvm_stdout.type = Console
appender.jvm_stdout.name = JVM_STDOUT
appender.jvm_stdout.layout.type = PatternLayout
appender.jvm_stdout.layout.pattern = %d{ISO8601} INFO jvm.stdout [%t] %m%n

logger.jvm_stdout.name = stdout
logger.jvm_stdout.level = INFO
logger.jvm_stdout.additivity = false
logger.jvm_stdout.appenderRef.jvm_stdout.ref = JVM_STDOUT

# 捕获JVM stderr并标记为INFO级别
appender.jvm_stderr.type = Console
appender.jvm_stderr.name = JVM_STDERR
appender.jvm_stderr.layout.type = PatternLayout
appender.jvm_stderr.layout.pattern = %d{ISO8601} INFO jvm.stderr [%t] %m%n

logger.jvm_stderr.name = stderr
logger.jvm_stderr.level = INFO
logger.jvm_stderr.additivity = false
logger.jvm_stderr.appenderRef.jvm_stderr.ref = JVM_STDERR

2. 在spark-submit时指定该配置文件：

spark-submit \
  --conf "spark.driver.extraJavaOptions=-Dlog4j.configurationFile=file:/path/to/your/log4j2.properties" \
  --conf "spark.executor.extraJavaOptions=-Dlog4j.configurationFile=file:/path/to/your/log4j2.properties" \
  # 其他spark-submit参数

这种方式会让JVM的所有标准输出/错误都通过Log4j发送，自动带上INFO级别标记。

内容的提问来源于stack exchange，提问作者svg_af_2