Spark计算Hive分区数据均值时为何存在不必要Shuffle？

Great question! Let's break down what's happening here, clarify why Spark's execution plan looks the way it does, and address the key misconception in your proposed approach.

First: Why Your "Per-Partition AVG + Union" Idea Doesn't Work #

Let’s get this critical point out of the way first: you can’t correctly compute a global or grouped average by merging per-partition averages directly. Here’s a quick example to prove it:

• Partition 1 has values [1, 3]: sum = 4, count = 2, AVG = 2
• Partition 2 has values [2, 4, 6]: sum = 12, count = 3, AVG = 4
• The correct global AVG is (4+12)/(2+3) = 3.2, but averaging the two partition AVGs gives (2+4)/2 = 3—which is wrong.

AVG is a non-additive aggregate function; you can’t merge AVG values directly. Instead, you need to track the additive building blocks (sum and count) from each partition, combine those globally, then compute the final AVG. That’s exactly what Spark is doing under the hood.

Why Spark's Execution Plan Includes Shuffle #

Let’s walk through Spark’s default strategy for calculating AVG:

1. Partial Aggregation: For each partition (or task), Spark first calculates the local sum and count of your target column. This happens entirely within each partition’s data—no shuffle needed here.
2. Shuffle (Exchange): If you’re computing a global AVG (no GROUP BY clause), or grouping by a column that isn’t the table’s partition key, Spark needs to bring all local sum/count pairs together to compute the total sum and total count. This requires a shuffle because the local aggregates from different partitions need to be processed in the same place.
3. Final Aggregation: Once all local sum/count values are gathered, Spark computes the total sum divided by total count to get the final AVG.

When Shuffle Is Unnecessary (And How to Fix It) #

If you’re grouping by the table’s partition column (e.g., SELECT date_, AVG(temp) FROM daily_temp.daily_temp_2014 GROUP BY date_), Spark should avoid shuffle entirely. Each partition’s local sum/count can be converted directly to the AVG for that date, with no need to shuffle data between partitions.

If you’re still seeing a shuffle in this scenario, try these fixes:

• Enable Adaptive Query Execution (AQE): Set spark.sql.adaptive.enabled=true (default in Spark 3+). AQE optimizes execution plans on the fly, including eliminating unnecessary shuffles for partition-aligned grouping.
• Refresh Partition Metadata: Run MSCK REPAIR TABLE daily_temp.daily_temp_2014 in Spark SQL to update the table’s partition info if it’s outdated. Spark can’t leverage partition alignment if it doesn’t have accurate metadata.
• Check Column Alignment: Ensure your GROUP BY column exactly matches the partition column (case-sensitive, same data type). Mismatches here will force Spark to shuffle to group data correctly.

Wrap-Up #

Spark’s default approach is actually the correct, efficient way to compute AVG. The shuffle you’re seeing is either necessary (for global or non-partition-aligned grouping) or a sign that an optimization isn’t enabled. Your intuition about per-partition processing is on the right track—Spark already does that for the additive sum/count components—but merging AVGs directly would lead to incorrect results.

内容的提问来源于stack exchange，提问作者pbahr