Spark Local与Standalone模式的精确差异及本地运行场景判定

Great question! Let’s dive into the exact differences between Spark’s Local and Standalone modes, covering every aspect you asked about, and wrap up with a clear answer for your laptop scenario.

1. Node/Machine Count #

• Local Mode: Strictly a single-machine setup. Every Spark component (Driver, Executor, internal services) runs within a single JVM process on one machine—no cluster or separate worker nodes involved.
• Standalone Mode: Spark’s built-in cluster manager, designed for scalable setups. It requires at least one Master node (to manage cluster resources) and one or more Worker nodes (to execute tasks). You can run it on a single machine (with separate Master and Worker processes), but it’s built to scale across multiple machines.

2. Memory & Core Allocation #

• Local Mode:
  • Memory: Driver and Executor share the single machine’s total memory. By default, the Driver uses 1GB of memory, and Executor resources are tied to the same process. You can tweak memory limits via spark.driver.memory in configs.
  • Core Count: You specify how many CPU cores to use (e.g., local[4] uses 4 cores, local[*] uses all available cores). All cores are from the single machine.
• Standalone Mode:
  • Memory: Master, Worker, and Executor processes have independent memory allocations. You set Worker node memory limits (via SPARK_WORKER_MEMORY in spark-env.sh), and Executors pull from this pool (e.g., --executor-memory 2g when submitting jobs). Driver memory is also configurable separately.
  • Core Count: Worker nodes have defined core limits (SPARK_WORKER_CORES), and you can assign specific core counts to Executors (e.g., --executor-cores 2). Total available cores are the sum of all Worker nodes’ cores.

3. Configuration & Deployment #

• Local Mode:
  • Zero deployment overhead—just install Spark and run commands directly.
  • No cluster management processes to start. You can launch Spark Shell or submit jobs with a simple --master local[*] flag (or let it default to Local Mode if no master is specified).
  • Configs are set via command-line arguments or spark-defaults.conf, but defaults work for most local testing.
• Standalone Mode:
  • Requires manual startup of cluster processes:

    # Start Master node
    ./sbin/start-master.sh
    # Start Worker node (replace <master-ip> with your master's IP/hostname)
    ./sbin/start-worker.sh spark://<master-ip>:7077
    

  • Cluster-wide configs are set in spark-env.sh (e.g., Master port, Worker resources). Jobs are submitted with --master spark://<master-ip>:7077.
  • Includes a web UI (default: http://<master-ip>:8080) to monitor cluster status, worker nodes, and running jobs.

4. Key Advantages #

• Local Mode:
  • Ultra-fast setup for development and debugging—no cluster to manage.
  • All logs are centralized in one process, making it easy to troubleshoot code issues.
  • Lightweight resource usage, perfect for testing small datasets or learning Spark APIs.
• Standalone Mode:
  • True cluster scalability—add more Worker nodes to handle larger datasets and heavier workloads.
  • Built-in resource scheduling to distribute tasks across nodes efficiently.
  • Supports cluster-mode job submission (Driver runs on the cluster instead of your local machine), making it suitable for production environments.

5.适用场景 (Use Cases) #

• Local Mode:
  • Writing and testing Spark code during development.
  • Running small-scale data analysis or proof-of-concept tasks.
  • Learning Spark fundamentals without the complexity of a cluster.
• Standalone Mode:
  • Production workloads for medium-scale datasets (when you don’t have access to YARN/Kubernetes).
  • Setting up a small team-based Spark cluster for shared data processing tasks.
  • Scenarios where you need cluster-level resource isolation and monitoring.

6. Examples #

Local Mode Example #

Launch Spark Shell using all available CPU cores:

./bin/spark-shell --master local[*]

Submit a compiled Spark job to Local Mode with 4 cores:

./bin/spark-submit --master local[4] --class com.example.MySparkApp my-app.jar

Standalone Mode Example #

First start the cluster:

# Start Master
./sbin/start-master.sh
# Start Worker (replace <master-ip> with your machine's IP)
./sbin/start-worker.sh spark://192.168.1.100:7077

Submit a job to the Standalone cluster with 2GB executor memory and 2 cores per executor:

./bin/spark-submit --master spark://192.168.1.100:7077 --executor-memory 2g --executor-cores 2 --class com.example.MySparkApp my-app.jar

Which Mode is Your Laptop Running? #

If you’re running Spark without explicitly starting a Standalone Master/Worker (e.g., just typing spark-shell or using --master local[*]), this is Local Mode.

Even if you start a Standalone Master and Worker on the same laptop, that counts as Standalone Mode—because you’re using separate cluster management processes. But in most cases, when people say "running Spark locally on my laptop", they mean Local Mode.

内容的提问来源于stack exchange，提问作者Nikhil Redij