在EMR集群运行Spark应用：配置指定位置及两种配置方式差异

Hey there! Let's break down your questions about Spark configuration on Amazon EMR clearly:

You have several options depending on the scope and flexibility you need:

• Cluster-wide default config file: Edit the spark/conf/spark-defaults.conf file (on EMR, the typical path is /etc/spark/conf/spark-defaults.conf). Any Spark app submitted to the cluster will inherit these settings by default—great for global, shared configurations.
• Per-app spark-submit command: Use the --conf flag when running your spark-submit command, like:

  spark-submit --conf spark.executor.instances=4 --conf spark.executor.memory=29G ... your-app.jar
  

  This applies only to the specific app you're submitting, perfect for one-off or app-specific tweaks.
• EMR cluster creation: When launching a new EMR cluster, you can define Spark configurations directly in the "Software Configuration" section. These settings will be automatically written to spark-defaults.conf when the cluster spins up, so you don't have to manually edit files later.
• Hardcoded in application code: Use the SparkConf object in your code (e.g., new SparkConf().set("spark.driver.memory", "12G")). While this has the highest priority, it's generally not recommended because it makes your code less flexible—you can't adjust settings without re-building the app.

Key Differences #

Let's break down the core distinctions:

• Scope:
  • spark-defaults.conf: Settings here apply to every Spark app running on the cluster. Ideal for universal defaults (like standard YARN memory overheads or parallelism) that all apps should use.
  • spark-submit: Configs added here only affect the single app you're submitting. Use this when you need to customize settings for a specific app without altering the cluster's global defaults.
• Priority:
  Spark follows a clear priority order for configuration values:
  Code-level SparkConf > spark-submit --conf > spark-defaults.conf > Spark's built-in defaults
  So if you set the same config key in both spark-defaults.conf and spark-submit, the spark-submit value will override the one in the config file.
• Maintainability:
  • spark-defaults.conf keeps your cluster's standard settings centralized—you set once, all apps use it. But modifying it requires updating the cluster's config files (or re-launching the cluster if you set it during creation), which affects all apps.
  • spark-submit gives you per-app flexibility, but you'll have to duplicate configs across multiple app submissions if they share the same requirements.

Are your example configurations equivalent? #

If no other configuration sources are overriding these values, then yes—setting all those parameters in spark-defaults.conf vs. passing them via spark-submit --conf flags will result in the exact same runtime configuration for your Spark app.

The only caveat: If the EMR cluster already has pre-existing settings in spark-defaults.conf that conflict with your example values, those would be overridden if you use spark-submit. But if you're starting with a clean slate, both methods will apply the same resource allocations and settings to your app.

内容的提问来源于stack exchange，提问作者Bilberryfm