Apache Spark 2.2.0 BlockManager内存计算疑问：YARN模式内存配置不符

Hey there! I’ve run into this exact confusion before, so let’s break it down clearly.

核心原因：配置的内存是总堆内存，BlockManager只用到其中一部分 #

When you set driver-memory or executor-memory to 1GB, that’s the total JVM heap memory allocated to the driver/executor. Spark doesn’t give all this memory to the BlockManager—it splits it into several distinct regions for different purposes.

Spark 2.2.0的BlockManager内存计算方式 #

Here’s the step-by-step breakdown of how BlockManager’s storage memory is calculated (using default parameters unless you’ve changed them):

• 总堆内存：Your configured driver-memory/executor-memory (1024MB in your case).
• 预留内存（Reserved Memory）：A fixed 300MB reserved for Spark’s internal metadata and system overhead. This is non-negotiable (unless you tweak a hidden test parameter, which isn’t recommended for production). If your total heap is less than 1.5GB, this stays at 300MB; for larger heaps, it adjusts to 1/6 of the total heap.
• 可用内存（Usable Memory）：Total Heap Memory - Reserved Memory → For your 1GB heap, that’s 1024MB - 300MB = 724MB.
• 统一内存池（Unified Memory Pool）：Spark allocates 60% (controlled by spark.memory.fraction, default 0.6) of the usable memory to a shared pool for execution (shuffles, joins, sorts) and storage (BlockManager caching). So 724MB × 0.6 ≈ 434.4MB.
• BlockManager存储内存：By default, 50% of the unified pool is reserved for storage (controlled by spark.memory.storageFraction, default 0.5). This is the base value registered by BlockManager—though execution memory can borrow from storage if needed, and vice versa.

为什么你的日志数值和默认计算有差异？ #

Your logs show 366.3MB for the driver and 414.4MB for the executor. Here are the most likely reasons:

• JVM实际堆内存偏差: Sometimes the JVM doesn’t allocate the exact heap size you specify (due to memory alignment or system constraints). For example, if your driver’s actual heap is closer to 1.2GB instead of 1GB, the calculation would line up with your log value.
• 自定义参数: You or your cluster admin might have modified spark.memory.fraction or spark.memory.storageFraction from their defaults. For instance, if spark.memory.fraction was set to 0.8 instead of 0.6, the unified pool would be larger, leading to a bigger storage memory value.
• Driver vs Executor细微差异: Drivers sometimes have slightly different memory overheads (like for UI threads or client-side processing) that can adjust the usable memory fraction.

如何验证？ #

To get to the bottom of it:

1. Check your Spark app’s Environment tab in the Spark UI—look for spark.memory.fraction and spark.memory.storageFraction to confirm their values.
2. Use jps to find the PID of your driver/executor process, then run jmap -heap <PID> to see the actual JVM heap size allocated.
3. Recalculate the storage memory using the actual heap size and the parameters from the UI—this should match the log value.

总结 #

The BlockManager memory you see in logs is just a slice of the total heap memory you configured. Spark splits the heap into reserved space, user code space, execution space, and storage space—so it’s completely normal for the BlockManager value to be much smaller than your 1GB setting.

内容的提问来源于stack exchange，提问作者Souvik Sarkhel