如何解决Azure Databricks交互式集群Driver因OOM频繁重启问题？

背景信息 #

• 因5分钟SLA要求，使用Databricks交互式集群
• 约24个作业在同一时间窗口（2-3分钟内）并发执行

我已附上集群RAM大小的截图（高亮了Driver重启的节点），问题如下：
为何Driver会累积内存并每隔n小时重启？我并未使用collect()、display()或将PySpark DataFrame转换为Pandas DataFrame的操作。

目前有4个交互式集群在运行，均出现类似问题，因为它们执行的作业/代码类似，仅参数不同。

据我所知，Driver应在作业完成后释放内存。还可以排查哪些内容来解决此问题？

标准错误日志 #

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:719)
at py4j.ClientServerConnection.startServerConnection(ClientServerConnection.java:110)
at py4j.JavaServer.createConnection(JavaServer.java:122)
at py4j.GatewayServer.processSocket(GatewayServer.java:680)
at py4j.GatewayServer.run(GatewayServer.java:706)
at py4j.GatewayServer.start(GatewayServer.java:789)
at com.databricks.backend.daemon.driver.PythonDriverLocalBase$Gateway.$anonfun$run$5(PythonDriverLocalBase.scala:814)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:1267)
at com.databricks.backend.daemon.driver.PythonDriverLocalBase$Gateway.$anonfun$run$3(PythonDriverLocalBase.scala:814)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.backend.daemon.driver.PythonDriverLocalBase$Gateway.retry(PythonDriverLocalBase.scala:819)
at com.databricks.backend.daemon.driver.PythonDriverLocalBase$Gateway.run(PythonDriverLocalBase.scala:808)

现在还看到以下错误：

java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:719)
    at zeromq_shaded.zmq.poll.Poller.start(Poller.java:181)
    at zeromq_shaded.zmq.Reaper.start(Reaper.java:58)
    at zeromq_shaded.zmq.Ctx.initSlots(Ctx.java:446)
    at zeromq_shaded.zmq.Ctx.createSocket(Ctx.java:388)
    at zeromq_shaded.org.zeromq.ZMQ$Socket.<init>(ZMQ.java:836)
    at zeromq_shaded.org.zeromq.ZMQ$Socket.<init>(ZMQ.java:822)
    at zeromq_shaded.org.zeromq.ZMQ$Context.socket(ZMQ.java:655)
    at zeromq_shaded.org.zeromq.ZContext.createSocket(ZContext.java:133)
    at com.databricks.backend.daemon.driver.JupyterClient.connect(JupyterClient.scala:252)
    at com.databricks.backend.daemon.driver.JupyterClient.<init>(JupyterClient.scala:258)
    at com.databricks.backend.daemon.driver.JupyterDriverLocal.$anonfun$startPython$1(JupyterDriverLocal.scala:698)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.util.Try$.apply(Try.scala:213)
    at com.databricks.backend.daemon.driver.JupyterDriverLocal.com$databricks$backend$daemon$driver$JupyterDriverLocal$$withRetry(JupyterDriverLocal.scala:663)
    at com.databricks.backend.daemon.driver.JupyterDriverLocal.startPython(JupyterDriverLocal.scala:680)
    at com.databricks.backend.daemon.driver.JupyterDriverLocal.<init>(JupyterDriverLocal.scala:403)
    at com.databricks.backend.daemon.driver.PythonDriverWrapper.instantiateDriver(DriverWrapper.scala:781)
    at com.databricks.backend.daemon.driver.DriverWrapper.setupRepl(DriverWrapper.scala:350)
    at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:246)
    at java.lang.Thread.run(Thread.java:750)

集群Ganglia内存图表 #

我已尝试增大Driver内存，但仅延迟了重启时间，未解决根本问题。

内容的提问来源于stack exchange，提问作者JKR