基于列对PySpark DataFrame执行repartition的效果及参数影响解析

Great question! Let’s break this down step by step so you get a clear picture of what’s happening under the hood.

When you run df.repartition('id') or df.repartition(num_partitions, 'id'), Spark uses hash partitioning based on the values of the specified column(s). Here’s the key behavior:

• All rows with the exact same id value will be moved to the same partition. This is guaranteed because Spark computes a hash of the id value, and rows with identical hashes are assigned to the same partition.
• The reverse isn’t true though: a single partition can contain rows with multiple different id values (unless you set the number of partitions equal to the number of unique id values, which is rarely practical).

For example, if you have id values 1, 2, 3, 4 and set 2 partitions, Spark might assign id=1 and id=3 to partition 0, and id=2 and id=4 to partition 1—but every row with id=1 will be in partition 0, no exceptions.

The spark.sql.shuffle.partitions parameter controls the default number of partitions for shuffle operations in Spark SQL. Its default value is 200, but how it interacts with repartition depends on how you call the method:

• If you use repartition without specifying a partition count (e.g., df.repartition('id')), Spark will use the value of spark.sql.shuffle.partitions as the target number of partitions. So with the default 200, your DataFrame will be split into 200 partitions, with id values hashed across them.
• If you explicitly specify a partition count (e.g., df.repartition(50, 'id')), the spark.sql.shuffle.partitions value is ignored—Spark will use the number you provided instead.
• One edge case to note: if the number of unique id values is less than the target partition count, some partitions will end up empty. For example, if you have 10 unique ids and target 200 partitions, only 10 partitions will have data, and the rest will be empty.

It’s also worth mentioning that this parameter affects other shuffle-based operations like groupBy, join, and orderBy, but explicit repartition calls always take precedence over the default.

内容的提问来源于stack exchange，提问作者Nikhil Baby