Spark 2.1（Cloudera环境）persist方法偶现无法完成问题求助

Absolutely, this is a reported issue that many Spark 2.1 users running on Cloudera have encountered when using persist(StorageLevel.MEMORY_AND_DISK()) on Datasets. Let’s break down the common causes and fixes based on community experiences:

1. Cloudera-specific Resource Configuration Mismatches #

Cloudera's default cluster settings (like memory allocation, YARN queue limits) often clash with Spark's caching needs. For example:

• If executor memory is too tight, Spark will try to spill data to disk—but local disk space shortages or high HDFS write latency can cause the persist operation to hang indefinitely.
• YARN container resource limits might lead to resource contention between your Spark app and other workloads on the cluster.

Fixes:

• Verify and adjust spark.executor.memory and spark.driver.memory allocations, ensuring executors have enough off-heap memory for disk caching.
• Check YARN queue resource quotas to make sure your app isn't starved for resources during persist operations.

2. Serialization Issues with Complex Dataset Types #

Spark 2.1's default Kryo serialization can have compatibility hiccups with complex data types (like custom Java/Scala objects, nested structs). Hidden serialization errors might not throw exceptions but will cause persist to stall silently.

Fixes:

• Explicitly configure Kryo serialization and register your custom classes to avoid unexpected behavior:

  spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  spark.conf.set("spark.kryo.registrationRequired", "true")
  spark.sparkContext.getConf.registerKryoClasses(Array(classOf[YourCustomDataType]))
  

• Temporarily switch to Java serialization (org.apache.spark.serializer.JavaSerializer) to rule out Kryo-specific bugs.

3. Corrupted Cache Block Metadata #

Occasionally, Spark's BlockManager can develop corrupted metadata for cached blocks—especially if cluster nodes restart unexpectedly or experience network blips. This leads to persist operations getting stuck while trying to validate or write cache blocks.

Fixes:

• Clear existing cache before persisting to avoid metadata conflicts:

  originalDataSet.unpersist()
  originalDataSet = originalDataSet.persist(StorageLevel.MEMORY_AND_DISK())
  

• Enable block management checks by setting spark.storage.blockManager.enableManagement to true—this helps detect and handle corrupted cache blocks automatically.

4. Cloudera Spark Patch Inconsistencies #

Cloudera distributes customized Spark 2.1 builds with their own patches. Early CDH 5.x versions had known bugs in the disk caching logic (like deadlocks during persist operations) that were fixed in later patches.

Fixes:

• Check your CDH version and upgrade to the latest available Spark patch for your Cloudera release. For example, CDH 5.13+ includes critical fixes for Spark 2.1 caching issues.

内容的提问来源于stack exchange，提问作者Firat Faruk EREN