You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Databricks上Spark任务出现无业务代码栈的NullPointerException排查求助

avatar
云上老周
2026-8-9
问题

在Databricks Runtime 11.3 LTS（基于Spark 3.3.0）上运行Scala编写的Spark任务，该任务涉及AWS S3与AWS Redshift的IO操作，执行时抛出不包含业务代码调用栈的NullPointerException，堆栈跟踪信息如下：

Job aborted due to stage failure: Task 267 in stage 30.0 failed 4 times, most recent failure: Lost task 267.3 in stage 30.0 (TID 6343) (10.218.226.53 executor 2): java.lang.NullPointerException
        at org.apache.spark.unsafe.map.BytesToBytesMap.safeLookup(BytesToBytesMap.java:534)
        at org.apache.spark.sql.execution.joins.UnsafeHashedRelation.get(HashedRelation.scala:231)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.hashAgg_doAggregateWithKeys_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:761)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:195)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$3(ShuffleMapTask.scala:81)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$1(ShuffleMapTask.scala:81)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.doRunTask(Task.scala:169)
        at org.apache.spark.scheduler.Task.$anonfun$run$4(Task.scala:137)
        at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:104)
        at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:137)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.Task.run(Task.scala:96)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:902)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1696)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:905)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:760)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
     
    Driver stacktrace:
    Caused by: NullPointerException: 
    at org.apache.spark.unsafe.map.BytesToBytesMap.safeLookup(BytesToBytesMap.java:534)
        at org.apache.spark.sql.execution.joins.UnsafeHashedRelation.get(HashedRelation.scala:231)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.hashAgg_doAggregateWithKeys_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:761)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:195)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$3(ShuffleMapTask.scala:81)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$1(ShuffleMapTask.scala:81)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.doRunTask(Task.scala:169)
        at org.apache.spark.scheduler.Task.$anonfun$run$4(Task.scala:137)
        at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:104)
        at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:137)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.Task.run(Task.scala:96)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:902)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1696)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:905)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:760)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
     
    Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3312)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:3244)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:3235)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:3235)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1424)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1424)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1424)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3524)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3462)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3450)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:51)
    Caused by: java.lang.NullPointerException
        at org.apache.spark.unsafe.map.BytesToBytesMap.safeLookup(BytesToBytesMap.java:534)
        at org.apache.spark.sql.execution.joins.UnsafeHashedRelation.get(HashedRelation.scala:231)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.hashAgg_doAggregateWithKeys_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:761)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:195)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$3(ShuffleMapTask.scala:81)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.ShuffleMapTask.$anonfun$runTask$1(ShuffleMapTask.scala:81)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.doRunTask(Task.scala:169)
        at org.apache.spark.scheduler.Task.$anonfun$run$4(Task.scala:137)
        at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:104)
        at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:137)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.scheduler.Task.run(Task.scala:96)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:902)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1696)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:905)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:760)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

请问排查这类问题的最佳方案是什么？

排查方案

1. 定位异常关联的业务逻辑环节

从堆栈信息可以看到，异常发生在Stage 30的ShuffleMapTask，涉及UnsafeHashedRelation（哈希连接）和hashAgg_doAggregateWithKeys（哈希聚合）操作。先通过Databricks UI查看该Stage对应的执行计划，找到任务中对应的join或聚合代码段，缩小排查范围。

2. 禁用全阶段代码生成获取业务代码栈

Spark的全阶段代码生成（Whole Stage Codegen）会将多个算子合并为动态生成的类，导致业务代码调用栈丢失。临时添加以下配置重新运行任务：

spark.conf.set("spark.sql.codegen.wholeStage", "false")

重新执行后，堆栈信息会包含具体的业务DataFrame操作调用栈，直接定位到触发NPE的代码位置。

3. 检查数据中的空值与非法记录

NPE发生在哈希表查找阶段，大概率是关联键/聚合键存在null值，或者IO过程中出现损坏数据：

  • 统计关键字段的null值占比：
    val keyNullStats = df.select(col("join_key").isNull.cast("int").alias("is_null"))
                     .agg(sum("is_null").alias("null_count"), count("*").alias("total_count"))
keyNullStats.show()
  • 验证S3文件完整性：检查任务读取的S3路径下是否有损坏的Parquet/CSV文件；查看Redshift导入导出日志，确认数据传输过程中是否有异常。

4. 排查Spark版本已知Bug

Spark 3.3.0存在多个与UnsafeHashedRelation和null值处理相关的已知Bug，比如哈希聚合时处理null键的逻辑缺陷。可以：

  • 查看Spark官方Bug列表，确认是否有对应问题的修复补丁；
  • 升级到Databricks Runtime 11.3 LTS的后续补丁版本，或直接迁移到13.x LTS版本（兼容性更好，修复了更多底层Bug）。

5. 调整配置规避底层问题

临时修改Spark配置，绕过可能触发NPE的底层逻辑：

  • 改用Sort Merge Join替代哈希连接：
    spark.conf.set("spark.sql.join.preferSortMergeJoin", "true")
  • 调整哈希表内存配置：增大spark.sql.shuffle.partitions减少每个Task处理的数据量，或开启堆外内存spark.memory.offHeap.enabled=true避免内存溢出导致的哈希表损坏。

6. 增加调试日志与本地复现

  • 在Databricks任务中开启DEBUG级日志，重点关注org.apache.spark.sql.execution.joins和org.apache.spark.unsafe包的日志，获取哈希表构建时的详细信息；
  • 抽取任务中的部分数据到本地开发环境，复现问题后逐步缩小范围，定位触发NPE的具体数据模式或操作组合。

内容的提问来源于stack exchange，提问作者alhuelamo

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.08.09 09:50:17