Spark 3.1.1+Operator环境下JMX Exporter无法暴露Driver指标排查

环境信息 #

• Spark版本：3.1.1
• Spark Operator版本：v1beta2-1.3.3-3.1.1

Spark Operator配置片段 #

monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    prometheus:
      jmxExporterJar: "/opt/spark/jars/jmx_prometheus_javaagent-0.11.0.jar"
      port: 8090
      configFile: "/opt/spark/work-dir/prometheus/prometheus.yaml"
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "2g"
    javaOptions: "-Dconfig.file=/opt/spark/work-dir/conf/application_app.conf -Dlog4j.configuration=file:///opt/spark/work-dir/log/log4j_app.properties"
    labels:
      version: 3.1.1
    serviceAccount: spark-jobs-spark

Prometheus JMX Exporter配置 #

# These come from the application driver if it's a streaming application
# Example: default/streaming.driver.com.example.ClassName.StreamingMetrics.streaming.lastCompletedBatch_schedulingDelay
- pattern: metrics<name=(\S+)\.(\S+)\.driver\.(\S+)\.StreamingMetrics\.streaming\.(\S+)><>Value
  name: spark_streaming_driver_$4
  labels:
    app_namespace: "$1"
    app_id: "$2"
- pattern: metrics<name=(\S+)\.(\S+)\.spark.streaming.(\S+)\.(\S+)><>Value
  name: streaming_query
  labels:
    job: "$1"
    instance: "$2"
    query: "$3"
    name: "$4"
# These come from the application driver if it's a structured streaming application
# Example: default/streaming.driver.spark.streaming.QueryName.inputRate-total
- pattern: metrics<name=(\S+)\.(\S+)\.driver\.spark\.streaming\.(\S+)\.(\S+)><>Value
  name: spark_structured_streaming_driver_$4
  labels:
    app_namespace: "$1"
    app_id: "$2"
    query_name: "$3"
# These come from the application executors
# Example: default/spark-pi.0.executor.threadpool.activeTasks
- pattern: metrics<name=(\S+)\.(\S+)\.(\S+)\.executor\.(\S+)><>Value
  name: spark_executor_$4
  type: GAUGE
  labels:
    app_namespace: "$1"
    app_id: "$2"
    executor_id: "$3"
# These come from the application driver
# Example: default/spark-pi.driver.DAGScheduler.stage.failedStages
- pattern: metrics<name=(\S+)\.(\S+)\.driver\.(BlockManager|DAGScheduler|jvm)\.(\S+)><>Value
  name: spark_driver_$3_$4
  type: GAUGE
  labels:
    app_namespace: "$1"
    app_id: "$2"
# These come from the application driver
# Emulate timers for DAGScheduler like messagePRocessingTime
- pattern: metrics<name=(\S+)\.(\S+)\.driver\.DAGScheduler\.(.*)><>Count
  name: spark_driver_DAGScheduler_$3_count
  type: COUNTER
  labels:

现象 #

应用Driver未暴露任何指标，但在Pod内使用spark-shell并指定javaagent配置JMX Exporter时可以正常获取指标，spark-submit方式则不行。

排查及解决方向 #

• 验证Java Agent参数注入情况
  查看Driver Pod的启动命令，确认是否包含-javaagent:/opt/spark/jars/jmx_prometheus_javaagent-0.11.0.jar=8090:/opt/spark/work-dir/prometheus/prometheus.yaml参数。如果缺失，检查Operator日志是否有配置加载错误，或确认exposeDriverMetrics开关是否正确生效。

• 手动添加Java Agent参数到javaOptions
  尝试在driver.javaOptions中手动加入javaagent参数（需放在参数最前面）：

  javaOptions: "-javaagent:/opt/spark/jars/jmx_prometheus_javaagent-0.11.0.jar=8090:/opt/spark/work-dir/prometheus/prometheus.yaml -Dconfig.file=/opt/spark/work-dir/conf/application_app.conf -Dlog4j.configuration=file:///opt/spark/work-dir/log/log4j_app.properties"
  

• 检查配置文件路径与权限
  进入Driver Pod，验证/opt/spark/work-dir/prometheus/prometheus.yaml文件存在且Spark进程有读取权限，执行：

  ls -l /opt/spark/work-dir/prometheus/
  cat /opt/spark/work-dir/prometheus/prometheus.yaml
  

• 确认JMX端口监听状态
  在Driver Pod内执行以下命令，检查8090端口是否被JMX Exporter监听：

  netstat -tulpn | grep 8090
  # 或使用ss命令
  ss -tulpn | grep 8090
  

  若未监听，说明javaagent未正确加载，需检查jar包是否损坏或配置文件格式错误。

• 排查Spark配置冲突
  在Driver Pod内通过spark-shell查看spark.driver.extraJavaOptions的实际值，确认是否有其他配置覆盖了Operator设置：

  sc.getConf.get("spark.driver.extraJavaOptions")
  

内容的提问来源于stack exchange，提问作者Alex L