为何MapReduce中间结果仅能落盘？为何无法像Spark一样存于内存？

Great question! This comes down to a mix of historical context, core design priorities, and architectural differences between the two frameworks. Let’s break it down:

1. Historical Hardware Constraints #

Back in 2004 when MapReduce was first introduced, server memory was astronomically expensive. A typical cluster node might only have 2-4GB of RAM—nowhere near enough to store the terabytes of intermediate data generated by large-scale jobs. Relying on memory for anything beyond tiny temporary buffers just wasn’t practical.

Spark, on the other hand, launched around 2012, by which time RAM had become cheap and abundant. Clusters could easily have nodes with 32GB+ of RAM, making in-memory caching a feasible and cost-effective choice.

2. Fault Tolerance by Design #

MapReduce was built for massive clusters where node failures are routine. Its fault tolerance model is disk-based for a simple reason: memory is volatile, but disk storage is persistent.

Every map task writes its output to local disk. If a node crashes mid-job, you only need to re-run the failed map task—its input is still on disk, and the new output will be written to disk again. There’s no dependency on transient in-memory data that would be lost if a node goes down.

Spark uses RDD lineage for fault tolerance: if an in-memory RDD partition is lost, it can be rebuilt by re-executing the sequence of transformations that created it. MapReduce doesn’t have this lightweight rebuild mechanism, so disk was the only reliable way to preserve intermediate results.

3. Strict Batch Pipeline Architecture #

MapReduce follows a rigid map → shuffle → reduce pipeline where all stages are sequential. All map tasks must complete before the shuffle phase starts, and all shuffle work must finish before reduce tasks run.

This means every map’s output has to be written to disk so the shuffle phase can scan, sort, and transfer all intermediate data across the cluster to the reduce nodes. There’s no way to pass data directly from map to reduce in memory because the stages are completely decoupled and run on potentially different nodes.

Spark’s RDD model allows for lazy evaluation and operation chaining. Intermediate results can stay in memory because subsequent transformations can directly reference them without needing to read from disk first. This creates a more flexible pipeline where data flows through memory across stages.

4. Task-Centric Resource Scheduling #

MapReduce uses a task-centric resource model: each map or reduce task gets allocated resources (CPU, memory) only for the duration of the task. Once the task finishes, those resources are immediately released back to the cluster.

There’s no persistent in-memory space across stages—once a map task ends, its memory is freed, so you can’t keep intermediate data around for the reduce phase.

Spark uses long-running Executors that hold onto memory throughout the job’s lifecycle. These Executors cache intermediate RDDs in memory, allowing subsequent stages to reuse the data without hitting disk.

内容的提问来源于stack exchange，提问作者Quan