Spark中Parallelised collections的概念、性能提升机制及分区配置问询

What are Parallelized Collections? #

Let's start with the basics: Parallelized Collections in Spark are a way to convert your local Scala/Java collections (like Array, List, or Set) into Resilient Distributed Datasets (RDDs). The core idea here is splitting your single-machine dataset into multiple smaller chunks called partitions, which are then distributed across different worker nodes in your cluster.

Think of it like slicing a big pizza into slices and passing each slice to a different friend to eat at the same time—instead of one person eating the whole pizza alone, everyone chows down in parallel.

How Do They Improve Job Performance? #

Parallelized collections are all about unlocking the power of distributed computing, and here's exactly how they make your Spark jobs faster and more efficient:

• True Parallel Processing: Instead of processing your dataset sequentially on a single machine, multiple worker nodes handle different partitions simultaneously. For large datasets, this cuts down processing time drastically—imagine processing 10GB of data on 8 nodes instead of 1.
• Maximized Resource Utilization: Without parallelization, most of your cluster's worker nodes would sit idle. Parallelized collections ensure every available CPU core and memory resource gets put to work, squeezing the most out of your hardware.
• Better Fault Tolerance: RDDs have built-in lineage (a record of how they're created), so if a worker node fails and loses a partition, Spark can recompute just that partition instead of reprocessing the entire dataset. This minimizes downtime and recovery costs.
• Lower Overhead for Local Data: If your source data is already in a local collection (e.g., loaded into the Driver node's memory), parallelizing it avoids the overhead of reading from external storage systems. It's a lightweight way to kick off distributed processing for small-to-medium datasets.

How to Configure Partitions for Parallelized Collections #

Partitioning is key to getting the most out of parallelized collections—too few partitions waste resources, too many add unnecessary scheduling overhead. Here are the main ways to set and adjust partitions:

1. Specify Partitions When Creating the RDD #

The simplest way is to pass a numPartitions parameter directly to the parallelize() method:

// Example in Scala
val localData = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
// Split the list into 4 partitions
val parallelRDD = sc.parallelize(localData, 4)

In Java, it works similarly:

List<Integer> localData = Arrays.asList(1,2,3,4,5,6,7,8,9,10);
JavaRDD<Integer> parallelRDD = sc.parallelize(localData, 4);

2. Use Default Cluster Configuration #

If you don't specify numPartitions, Spark falls back to the default value set by spark.default.parallelism. For Standalone or YARN clusters, this is typically the total number of available CPU cores across all worker nodes. For local mode (e.g., local[*]), it uses the number of cores on your machine.

3. Adjust Partitions After RDD Creation #

If you need to change the number of partitions post-creation, use one of these methods:

• repartition(numPartitions): This shuffles the data across the cluster to create the desired number of partitions. Use this when you need to increase the number of partitions, or when you need a radical reduction that can't be done without shuffling.

  val rePartitionedRDD = parallelRDD.repartition(6)
  

• coalesce(numPartitions): This method avoids shuffling whenever possible (by merging existing partitions), making it more efficient for reducing the number of partitions. Great for cases where you have too many small partitions and want to consolidate them.

  val coalescedRDD = parallelRDD.coalesce(2)
  

4. Set a Global Default for Your Application #

You can configure the default parallelism for your entire Spark app by setting the spark.default.parallelism property in your SparkConf:

val conf = new SparkConf()
  .setAppName("MyParallelApp")
  .setMaster("yarn")
  .set("spark.default.parallelism", "8") // Default to 8 partitions
val sc = new SparkContext(conf)

Quick Tip on Partition Size #

As a general rule of thumb, aim for partitions that are between 100MB and 1GB in size. This balances parallelism and scheduling overhead—smaller partitions lead to more task scheduling work, while larger partitions can leave some nodes idle.

内容的提问来源于stack exchange，提问作者HungryBird