如何调整Spark配置解决Databricks集群5秒Future超时问题

集群配置 #

Driver: Standard_E8ds_v5 
Workers: Standard_E8ds_v5
30 workers 
11.3 LTS Photon (includes Apache Spark 3.3.0, Scala 2.12)

问题情况 #

作业通过Python线程池并行调用子Notebook执行任务，约5%的场景会触发Futures timed out after [5 seconds]错误，此时集群因大量并行操作处于繁忙状态，子Notebook的多个Spark代码段都可能出现执行失败。

作业执行逻辑 #

def RunChild(s):
  dbutils.notebook.run("./ProcessChild", 0, {"param": s})

scenarios = [ some array with 107 items]
with ThreadPoolExecutor(max_workers=20) as executor:
   final = executor.map(RunChild, scenarios)

错误栈跟踪 #

java.util.concurrent.TimeoutException: Futures timed out after [5 seconds] 
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
        at scala.concurrent.Await$.$anonfun$result$1(package.scala:223)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
        at scala.concurrent.Await$.result(package.scala:146)
        at com.databricks.backend.daemon.driver.JupyterDriverLocal$RequestStatus.waitForReply(JupyterDriverLocal.scala:209)
        at com.databricks.backend.daemon.driver.JupyterDriverLocal.repl(JupyterDriverLocal.scala:971)
        at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$23(DriverLocal.scala:725)
        at com.databricks.unity.EmptyHandle$.runWith(UCSHandle.scala:103)
        at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$20(DriverLocal.scala:708)
        at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:398)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:147)
        at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:396)
        at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:393)
        at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:62)
        at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:441)
        at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:426)
        at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:62)
        at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:685)
        at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:622)
        at scala.util.Try$.apply(Try.scala:213)
        at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:614)
        at com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:533)
        at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:568)
        at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:438)
        at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:381)
        at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:232)
        at java.lang.Thread.run(Thread.java:750)

配置调整方案 #

从栈跟踪可以定位到，超时是Driver处理REPL请求时的等待超时导致的，对应的Spark配置项为spark.databricks.repl.awaitTimeout，默认值为5000毫秒（即5秒）。

在集群的Spark配置中添加以下参数即可延长超时时间，示例设置为30秒：

spark.databricks.repl.awaitTimeout 30000

该配置控制Driver等待REPL请求响应的最长时间，集群繁忙时调高此值，能避免因请求响应延迟触发的超时错误。

内容的提问来源于stack exchange，提问作者GregGalloway