Spark配置需在JAR加载前应用吗？类加载异常及配置报错求助

我创建了一个包含NDBSparkSessionExtension的JAR文件，通过以下命令验证类存在：

> jar -tf vast-uber.jar | grep ndb.NDBSparkSessionExtension
ndb/NDBSparkSessionExtension.class

我已将该JAR上传至UC存储卷，并添加为集群库。

集群配置 #

spark.ndb.secret_access_key ****
spark.ndb.rowgroups_per_subsplit 1
spark.ndb.access_key_id ****
spark.sql.extensions ndb.NDBSparkSessionExtension
spark.ndb.dynamic_filter_compaction_threshold 100
spark.sql.catalog.ndb spark.sql.catalog.ndb.VastCatalog
spark.ndb.retry_sleep_duration 1
spark.ndb.endpoint http://172.19.197.1
spark.ndb.retry_max_count 3
spark.ndb.parallel_import true
spark.master local[*, 4]
spark.ndb.num_of_sub_splits 8
spark.databricks.cluster.profile singleNode
spark.ndb.num_of_splits 64
spark.ndb.data_endpoints http://127.0.0.1
spark.ndb.query_data_rows_per_split 4000000
spark.ndb.dynamic_filtering_wait_timeout 2

日志报错信息 #

集群启动日志显示，SparkContext初始化后才开始加载JAR，导致SparkSession初始化时找不到扩展类：

25/02/06 10:41:43 INFO DatabricksILoop$: Successfully initialized SparkContext
25/02/06 10:41:43 WARN SparkSession: Cannot use ndb.NDBSparkSessionExtension to configure session extensions.
java.lang.ClassNotFoundException: ndb.NDBSparkSessionExtension
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    ...
25/02/06 10:42:07 INFO DriverCorral: [Thread 170] AttachLibraries - candidate libraries: List(dbfs:/Volumes/databricks_catalog/default/vastjars/vast-uber.jar)
25/02/06 10:42:07 INFO LibraryDownloadManager: Preparing to download library file from UC Volume path: dbfs:/Volumes/databricks_catalog/default/vastjars/vast-uber.jar

我的疑问是：我原本期望库会在应用配置之前加载，请问Spark配置是否会在JAR文件加载之前生效？

使用的Databricks Runtime版本：15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)

我尝试在集群配置中添加spark.jars dbfs:/Volumes/databricks_catalog/default/vastjars/vast-uber.jar，出现以下错误：

25/02/06 11:09:22 WARN FileSystem: Failed to initialize filesystem dbfs:/Volumes/databricks_catalog/default/vastjars/vast-uber.jar: com.databricks.backend.daemon.data.common.InvalidMountException: Error while using path /Volumes/databricks_catalog/default/vastjars for creating file system within mount at '/Volumes/databricks_catalog/default/vastjars'.
25/02/06 11:09:22 ERROR SparkContext: Failed to add dbfs:/Volumes/databricks_catalog/default/vastjars/vast-uber.jar to Spark environment
com.databricks.backend.daemon.data.common.InvalidMountException: Error while using path /Volumes/databricks_catalog/default/vastjars for creating file system within mount at '/Volumes/databricks_catalog/default/vastjars'.
    at com.databricks.backend.daemon.data.common.InvalidMountException$.apply(DataMessages.scala:765)
    at com.databricks.backend.daemon.data.filesystem.MountEntryResolver.createFileSystem(MountEntryResolver.scala:154)

如果去掉dbfs:前缀，使用配置spark.jars /Volumes/databricks_catalog/default/vastjars/vast-uber.jar，集群直接无法启动。

内容的提问来源于stack exchange，提问作者Chris Snow