Spark-submit中num-executors能否覆盖已启用的动态资源分配？

Short answer: No—when dynamic resource allocation (DRA) is enabled (spark.dynamicAllocation.enable=true), specifying --num-executors in your spark-submit command will not override the DRA behavior. In fact, Spark will completely ignore the num-executors parameter in this scenario.

Here's why: #

Dynamic allocation is built to automatically scale the number of executors based on your application's workload (adding executors when there's pending work, removing them when they're idle). When enabled, Spark uses three dedicated configuration parameters to manage executor counts instead of spark.executor.instances (which is what --num-executors sets):

• spark.dynamicAllocation.minExecutors: The minimum number of executors that will remain running at all times.
• spark.dynamicAllocation.maxExecutors: The upper limit for how many executors Spark can scale up to.
• spark.dynamicAllocation.initialExecutors: The number of executors launched when your application starts (defaults to minExecutors if not explicitly set).

Example scenario: #

If you run a command like this:

spark-submit \
  --num-executors 15 \
  --conf spark.dynamicAllocation.enable=true \
  --conf spark.dynamicAllocation.minExecutors=2 \
  --conf spark.dynamicAllocation.maxExecutors=8 \
  your-app.jar

Spark will start with 2 executors, scale up to a maximum of 8 as needed, and completely ignore the --num-executors 15 value.

What if you want fixed executors? #

If you need a static number of executors, you must disable dynamic allocation first by setting spark.dynamicAllocation.enable=false. Only then will --num-executors take effect and lock in the exact number of executors for your application.

Alternatively, if you want to mimic a fixed count while keeping DRA enabled (though this defeats its auto-scaling purpose), you can set minExecutors and maxExecutors to the same value. This will keep the executor count static, but it's not the same as using --num-executors.

内容的提问来源于stack exchange，提问作者Arvind Kumar