使用df.cache()后是否必须调用df.unpersist()释放缓存内存？

Hey, great question—let’s break this down clearly so you know exactly what’s going on here.

是否必须调用df.unpersist()来释放缓存内存？ #

Short answer: No, you don’t have to. Spark comes with a built-in cache management system that uses an LRU (Least Recently Used) policy to handle memory automatically. When your cluster runs low on available memory, Spark will automatically evict cached data that hasn’t been accessed in a while to make room for new jobs or cached datasets.

That said, manually calling df.unpersist() is still useful in specific cases. If you’re 100% sure a cached DataFrame won’t be used again for the rest of your workflow, explicitly freeing up its memory can prevent resource waste—especially if you’re working with huge datasets or running back-to-back jobs where memory is tight. It helps lower the risk of out-of-memory errors (OOM) down the line.

为什么不调用unpersist()速度快，调用后耗时明显变长？ #

This is totally expected behavior, and it boils down to what caching actually does. When you call df.cache(), Spark will store the computed result of that DataFrame in memory (or whatever storage level you specify, like memory + disk) the first time it’s calculated. Every subsequent time you use that DataFrame, Spark skips re-running all the expensive transformations (filters, joins, aggregations, etc.) and just pulls the pre-computed data from cache—hence the speed boost.

When you call df.unpersist(), you’re telling Spark to immediately delete that cached data. Now, any time you need that DataFrame again, Spark has to re-execute every single step in its lineage to recompute the data from scratch. That’s why you see a big jump in runtime.

Quick tip: #

Only call unpersist() when you’re done with the DataFrame for good. If you plan to reuse it multiple times in your job, leaving it cached will save you a ton of computation time.

内容的提问来源于stack exchange，提问作者Markus