DataProcCluster中spark-submit运行Java Spark应用失败排查

问题描述 #

在Spark 2.4.8版本的DataProc集群上运行Java Spark应用，本地项目Spark版本同样为2.4.8，应用尝试运行2次后失败。核心错误为初始化SparkContext时触发IllegalArgumentException："Can only call getServletHandlers on a running MetricsSystem"，已显式添加MetricsSystem相关代码但问题仍未解决。

错误日志 #

Application application_1676252836684_0013 failed 2 times due to AM Container for appattempt_1676252836684_0013_000002 exited with exitCode: 13
Failing this attempt.Diagnostics: [2023-02-13 08:39:33.239]Exception from container-launch.
Container id: container_1676252836684_0013_02_000001
Exit code: 13
[2023-02-13 08:39:33.240]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
onnected to master
23/02/13 08:39:32 ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
23/02/13 08:39:32 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
23/02/13 08:39:32 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:470)
at org.apache.spark.deploy.yarn.ApplicationMaster.runImpl(ApplicationMaster.scala:305)
at org.apache.spark.deploy.yarn.ApplicationMaster.$anonfun$run$1(ApplicationMaster.scala:245)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:781)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:780)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:805)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
[2023-02-13 08:39:33.240]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
onnected to master
23/02/13 08:39:32 ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
23/02/13 08:39:32 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
23/02/13 08:39:32 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:470)
at org.apache.spark.deploy.yarn.ApplicationMaster.runImpl(ApplicationMaster.scala:305)
at org.apache.spark.deploy.yarn.ApplicationMaster.$anonfun$run$1(ApplicationMaster.scala:245)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:781)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:780)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:805)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
For more detailed output, check the application tracking page: http://spark-cluster-m:8188/applicationhistory/app/application_1676252836684_0013 Then click on links to logs of each attempt.
. Failing the application.

现有Spark配置代码 #

.set("spark.submit.deployMode", "cluster").set("spark.driver.maxResultSize", "6g")
.set("spark.executor.memory", "6g").setMaster("spark://35.194.176.13:8188").set("spark.cores.max", "8")
.set("spark.blockManager.port", "10025").set("spark.driver.blockManager.port", "10026")
.set("spark.driver.port", "10027").set("spark.shuffle.service.enabled", "false")
.set("spark.dynamicAllocation.enabled", "false");
JavaSparkContext jscontext = new JavaSparkContext(config);
SecurityManager securityMgr = new SecurityManager(config, null, null);
MetricsSystem metricsSystem = MetricsSystem.createMetricsSystem("GraphNetworkSCLAlgorithm", config,
        securityMgr);
metricsSystem.start(true);

错误原因分析 #

1. 集群模式与Master配置不匹配：DataProc集群默认基于YARN作为资源管理器，但配置中指定了spark://的Standalone模式Master地址，同时设置deployMode=cluster，导致SparkContext初始化流程混乱，内部MetricsSystem无法正常启动。
2. 手动创建MetricsSystem干扰内部逻辑：SparkContext初始化时会自动创建并启动专属的MetricsSystem，在SparkContext创建后手动创建的额外MetricsSystem不仅无效，还会导致内部调用getServletHandlers时，对应MetricsSystem状态异常。
3. 不必要的端口手动配置：YARN Cluster模式下，Spark会自动分配所需端口，手动指定blockManager.port、driver.port等端口可能引发端口冲突或绑定失败，进一步干扰MetricsSystem启动。

解决方法 #

1. 修正Master配置 #

将setMaster("spark://35.194.176.13:8188")替换为setMaster("yarn")，适配DataProc的YARN集群环境：

.setMaster("yarn")

2. 移除手动创建的MetricsSystem代码 #

删除以下无关代码，依赖SparkContext自动初始化的MetricsSystem：

SecurityManager securityMgr = new SecurityManager(config, null, null);
MetricsSystem metricsSystem = MetricsSystem.createMetricsSystem("GraphNetworkSCLAlgorithm", config,
        securityMgr);
metricsSystem.start(true);

3. 移除不必要的端口配置 #

删除所有手动指定端口的配置项，让YARN自动分配端口：

// 移除以下配置
.set("spark.blockManager.port", "10025").set("spark.driver.blockManager.port", "10026")
.set("spark.driver.port", "10027")

修正后的完整配置示例 #

SparkConf config = new SparkConf()
        .set("spark.submit.deployMode", "cluster")
        .set("spark.driver.maxResultSize", "6g")
        .set("spark.executor.memory", "6g")
        .setMaster("yarn")
        .set("spark.cores.max", "8")
        .set("spark.shuffle.service.enabled", "false")
        .set("spark.dynamicAllocation.enabled", "false");
JavaSparkContext jscontext = new JavaSparkContext(config);

内容的提问来源于stack exchange，提问作者Aavash Bhandari