使用YARN+公平调度器，能否通过spark-submit设置spark.scheduler.pool本地属性？

No, you can't set the spark.scheduler.pool local property directly using spark-submit with -Dspark.scheduler.pool=xxx (or --conf)—here's why, plus workarounds to achieve similar behavior:

Why this doesn't work #

The spark.scheduler.pool is a SparkContext local property, not a global Spark configuration setting. The core distinction is:

• Global configs set via spark-submit are loaded when the SparkContext initializes, applying to the entire application context.
• Local properties like spark.scheduler.pool are meant to be set after the SparkContext is created, and are scoped to individual threads or jobs. This design lets you route different jobs from the same application into separate Fair Scheduler pools.

Workarounds to control the pool via spark-submit #

If you want to pass the pool name at submission time, use this approach:

1. Add a custom global config parameter in your spark-submit command:

   spark-submit --conf spark.custom.scheduler.pool=myPool your-application.jar
   

2. In your code, read this custom config and set the local property:

   val sc: SparkContext = ???
   // Read the custom config, with a default fallback if not provided
   val targetPool = sc.getConf.get("spark.custom.scheduler.pool", "default")
   sc.setLocalProperty("spark.scheduler.pool", targetPool)
   

This lets you control the pool name at submission while respecting the local property's intended design. If you want all jobs in your app to use the same pool, set this property once right after initializing the SparkContext—all subsequent jobs will inherit the setting.

内容的提问来源于stack exchange，提问作者Ashwini