Spark执行内存、存储内存及用户内存监控方法问询

Great question! Let's break down how to monitor both execution memory and the often-overlooked user memory in Spark, aligned with its official memory management model:

一、监控执行内存使用率 #

Execution memory is Spark's dedicated space for computation tasks like shuffles, joins, and sorting. Here are reliable ways to track it:

• Spark UI (Real-Time Check)
  Head to the Executors tab in your Spark UI—each executor entry shows Used Execution Memory and Total Execution Memory, so you can calculate usage rate directly. For task-level details, dive into individual Stage pages under the Jobs tab, where you'll find execution memory stats per task.

• Metrics System for Long-Term Monitoring
  If you need to integrate with tools like Prometheus, enable Spark's Metrics system and configure the appropriate sink. Key metrics to track:

  • executor.memory.execution.used: Current execution memory consumption
  • executor.memory.execution.total: Total allocated execution memory
    Compute usage rate with: (used / total) * 100%

• Custom Code Tracking
  You can fetch execution memory stats directly in your application code:

  // Driver-side: Sum execution memory across all executors
  val totalExecutionMemoryUsed = sc.getExecutorMemoryStatus.values.map(_._2).sum
  
  // Task-side: Get memory usage for the current task
  val taskExecutionMemory = TaskContext.get().taskMetrics().executionMemoryUsage
  

二、监控用户内存（非执行/非存储内存） #

User memory refers to the heap space not allocated to execution or storage memory (and outside the scope of spark.memory.fraction). It's used for user-defined data structures, UDFs, and other custom code objects. Here's how to monitor it:

• Calculate via Executor Heap Metrics
  User memory can be derived by subtracting other memory components from the total executor heap:

  User Memory Usage = Total Executor Heap - Execution Memory Used - Storage Memory Used - Reserved Memory (default 300MB)
  You can compute this either:

  1. Manually using values from the Spark UI's Executors tab (Total Heap Memory, Used Execution Memory, Used Storage Memory);
  2. Via Metrics: Subtract executor.memory.execution.used and executor.memory.storage.used from executor.memory.heap.used, then subtract the default 300MB reserved memory.

• JVM-Level Monitoring
  Since user memory lives in the JVM heap, use tools like jstat or jmap to inspect heap allocation details. You can also expose JMX metrics (e.g., java.lang:type=Memory's HeapMemoryUsage) and combine them with Spark's execution/storage metrics to isolate user memory usage.

• Code-Level Tracking
  In your custom code (UDFs, custom operators), track JVM memory and compute user memory:

  val runtime = Runtime.getRuntime()
  val totalHeap = runtime.totalMemory()
  val freeHeap = runtime.freeMemory()
  val usedHeap = totalHeap - freeHeap
  
  // Assume executionUsed and storageUsed are fetched from Spark metrics
  val reservedMemory = 300 * 1024 * 1024 // Default 300MB
  val userMemory = usedHeap - executionUsed - storageUsed - reservedMemory
  

Quick Note on Spark Memory Boundaries #

Remember: spark.memory.fraction only controls the split between execution and storage memory (after subtracting reserved memory). User memory sits outside this split, so it's entirely managed by your application code—keep an eye on it to avoid unexpected OOM errors from custom data structures.

内容的提问来源于stack exchange，提问作者Antonio Ye