Spark写入Hadoop最佳实践：按分区列重分区再写入是否合理？

Great question! Let’s unpack whether calling repartition on your target partition columns before using partitionBy to write data is a one-size-fits-all best practice—it depends heavily on your specific data and workload, so it’s not universally recommended.

When Pre-Repartitioning Makes Sense #

• Mitigate small file issues: If your source data has an uneven distribution relative to your target partition columns, pre-repartitioning can help control the number of files written per partition. For example, if your raw data is split into hundreds of unaligned partitions, repartition(partitionCols) will shuffle data so that each target partition maps to a predictable number of files (governed by spark.sql.shuffle.partitions), avoiding a flood of tiny files that hurt read performance later.
• Consistent partitioning for downstream operations: If you plan to perform additional transformations on the data after repartitioning but before writing, aligning partitions early can avoid repeated shuffles in subsequent steps.

When Pre-Repartitioning Is Unnecessary (Or Harmful) #

• Source data is already partitioned by the target columns: If you’re reading data that’s already partitioned on partitionCols (e.g., from a pre-existing partitioned Parquet table), adding an explicit repartition triggers an unnecessary shuffle. Spark will automatically leverage the existing partitioning when using partitionBy to write, so you can skip this step entirely to save resources.
• High-cardinality partition columns: If partitionCols have extremely high cardinality (e.g., partitioning by user ID with millions of unique values), repartition will create a huge number of shuffle partitions. This can overwhelm Executors with memory pressure or spawn thousands of small tasks, slowing down the job instead of optimizing it.

Recommended Best Practices #

1. Assess your source data first: Check if the input DataFrame is already partitioned on your target columns (use df.rdd.getNumPartitions and inspect partition distribution). If it’s aligned, skip repartition.
2. Control shuffle partition count: If you do use repartition, explicitly set a reasonable number of partitions instead of relying on the default spark.sql.shuffle.partitions (which is 200). For example:

   df.repartition(100, partitionCols.map(col): _*) // Adjust 100 based on data size
   

3. Let Spark handle it when possible: If you don’t need to enforce partition count upfront, Spark will automatically shuffle data during the partitionBy write. This is often sufficient unless you’re actively trying to fix file size issues.

内容的提问来源于stack exchange，提问作者user3685285