使用ipywidgets加载Databricks Unity Catalog表抛出"Missing Credential Scope"异常
2026-8-5
问题
我们尝试结合ipywidgets创建数据处理Notebook，为预定义函数调用搭建简易UI，过程中需要从Metastore加载数据。示例代码如下：
def loaddf(): df = spark.read.table('sandbox.mysamples.sampledata') return df def clickButton(b): output.clear_output() with output: df = loaddf() return ... search_btnOne = Button( .... ) search_btnOne.on_click(clickButton)
其中，loaddf函数用于加载DataFrame，clickButton函数调用该函数供下游使用，ipywidgets按钮点击时触发clickButton函数。直接在代码单元格中调用loaddf()可正常加载数据并通过.display()展示，但点击按钮时会抛出如下错误：
File <command-2062771599631884>:8, in clickButton(b) 6 output.clear_output() 7 with output: ----> 8 df = loaddf() 9 return File <command-2062771599631884>:2, in loaddf() 1 def loaddf(): ----> 2 df = spark.read.table('sandbox.mysamples.sampledata') 3 return df File /databricks/spark/python/pyspark/instrumentation_utils.py:48, in _wrap_function.<locals>.wrapper(*args, **kwargs) 46 start = time.perf_counter() 47 try: ---> 48 res = func(*args, **kwargs) 49 logger.log_success( 50 module_name, class_name, function_name, time.perf_counter() - start, signature 51 ) 52 return res File /databricks/spark/python/pyspark/sql/readwriter.py:320, in DataFrameReader.table(self, tableName) 303 def table(self, tableName: str) -> "DataFrame": 304 """Returns the specified table as a :class:`DataFrame`. 305 306 .. versionadded:: 1.4.0 (...) 318 [('name', 'string'), ('year', 'int'), ('month', 'int'), ('day', 'int')] 319 """ ---> 320 return self._df(self._jreader.table(tableName)) File /databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args) 1315 command = proto.CALL_COMMAND_NAME +\ 1316 self.command_header +\ 1317 args_command +\ 1318 proto.END_COMMAND_PART 1320 answer = self.gateway_client.send_command(command) -> 1321 return_value = get_return_value( 1322 answer, self.gateway_client, self.target_id, self.name) 1324 for temp_arg in temp_args: 1325 temp_arg._detach() File /databricks/spark/python/pyspark/sql/utils.py:196, in capture_sql_exception.<locals>.deco(*a, **kw) 194 def deco(*a: Any, **kw: Any) -> Any: 195 try: ---> 196 return f(*a, **kw) 197 except Py4JJavaError as e: 198 converted = convert_exception(e.java_exception) File /databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE: ---> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value)) Py4JJavaError: An error occurred while calling o10344.table. : org.apache.spark.SparkException: Missing Credential Scope. at com.databricks.unity.UCSDriver$Manager.$anonfun$scope$1(UCSDriver.scala:103) at scala.Option.getOrElse(Option.scala:189) at com.databricks.unity.UCSDriver$Manager.scope(UCSDriver.scala:103) at com.databricks.unity.UCSDriver$Manager.currentScope(UCSDriver.scala:97) at com.databricks.unity.UnityCredentialScope$.currentScope(UnityCredentialScope.scala:100) at com.databricks.unity.UnityCredentialScope$.getCredentialManager(UnityCredentialScope.scala:128) at com.databricks.unity.CredentialManager$.getUnityApiTokenOpt(CredentialManager.scala:456) at com.databricks.unity.UnityCatalogClientHelper$.getToken(UnityCatalogClientHelper.scala:35) at com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$bulkGetMetadata$1(ManagedCatalogClientImpl.scala:2889) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapException$1(ManagedCatalogClientImpl.scala:2953) at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException(ErrorDetailsHandler.scala:25) at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException$(ErrorDetailsHandler.scala:23) at com.databricks.managedcatalog.ManagedCatalogClientImpl.wrapServiceException(ManagedCatalogClientImpl.scala:79) at com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapException(ManagedCatalogClientImpl.scala:2952) at com.databricks.managedcatalog.ManagedCatalogClientImpl.bulkGetMetadata(ManagedCatalogClientImpl.scala:2882) at com.databricks.sql.managedcatalog.NonPermissionEnforcingManagedCatalog.updateCache(NonPermissionEnforcingManagedCatalog.scala:49) at com.databricks.sql.managedcatalog.PermissionEnforcingManagedCatalog.getTablesByName(PermissionEnforcingManagedCatalog.scala:244) at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.fastGetTablesByName(ManagedCatalogSessionCatalog.scala:1061) at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.fetchFromCatalog(DeltaCatalog.scala:417) at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$loadTables$1(DeltaCatalog.scala:362) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:248) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:246) at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:80) at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.loadTables(DeltaCatalog.scala:359) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anon$3.$anonfun$submit$1(Analyzer.scala:1819) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$record(Analyzer.scala:1878) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anon$3.submit(Analyzer.scala:1801) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1430) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1370) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$4(RuleExecutor.scala:218) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$3(RuleExecutor.scala:218) at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) at scala.collection.immutable.List.foldLeft(List.scala:91) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:215) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeBatch$1(RuleExecutor.scala:207) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$8(RuleExecutor.scala:277) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$8$adapted(RuleExecutor.scala:277) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:277) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:194) at org.apache.spark.sql.catalyst.analysis.Analyzer.executeSameContext(Analyzer.scala:353) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:346) at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:253) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:346) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:274) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:186) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:153) at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:186) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:326) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:331) at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:325) at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:163) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:319) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$3(QueryExecution.scala:353) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:789) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:353) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1003) at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:350) at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:144) at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:144) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:136) at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:98) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1003) at org.apache.spark.sql.SparkSession.$anonfun$withActiveAndFrameProfiler$1(SparkSession.scala:1010) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80) at org.apache.spark.sql.SparkSession.withActiveAndFrameProfiler(SparkSession.scala:1010) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) at org.apache.spark.sql.DataFrameReader.table(DataFrameReader.scala:802) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380) at py4j.Gateway.invoke(Gateway.java:306) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195) at py4j.ClientServerConnection.run(ClientServerConnection.java:115) at java.lang.Thread.run(Thread.java:750)
相关情况说明：
- 该问题在共享集群和单用户集群中均会出现；
- MANAGED表和EXTERNAL表均受此问题影响；
- 启用Unity Catalog前使用hive-metastore时未出现该问题；
- 用户拥有存储凭证、外部位置（针对EXTERNAL表）及所用目录的完全权限。
原因分析与解决方案
原因
这个问题的核心是ipywidgets的回调函数运行在独立线程中，而Unity Catalog的凭证作用域（Credential Scope）没有被正确传递到该线程。
在Notebook单元格直接执行代码时，Spark会话和Unity Catalog的凭证上下文绑定在当前主线程中，所有操作都能正常获取到凭证信息。但ipywidgets的点击回调是在单独的后台线程触发的，这个线程没有继承主线程的Unity Catalog凭证作用域，导致访问Unity Catalog中的表时，无法获取必要的认证信息，从而抛出Missing Credential Scope错误。
解决方案
可以通过以下几种方式解决：
方式一：在回调函数中显式激活Spark会话
修改clickButton函数，在调用loaddf前，通过spark.session()上下文管理器绑定主线程的Spark会话上下文：
def clickButton(b): output.clear_output() with output: # 显式激活当前Spark会话上下文 with spark.session(): df = loaddf() # 后续处理逻辑，比如展示数据 display(df)
方式二：使用dbutils.notebook.run封装加载逻辑
将加载数据的逻辑放到单独的Notebook中，然后在回调函数中通过dbutils.notebook.run调用，这样会在主线程上下文中执行，自动携带Unity Catalog的凭证作用域：
# 新建一个名为load_data_notebook的Notebook，内容为： # df = spark.read.table('sandbox.mysamples.sampledata') # display(df) def clickButton(b): output.clear_output() with output: dbutils.notebook.run("load_data_notebook", 60)
方式三：手动绑定主线程的Spark会话
在主线程中获取当前活跃的Spark会话，然后在回调函数中手动设置为活跃会话：
from pyspark.sql import SparkSession # 在主线程中获取当前活跃的Spark会话 current_session = SparkSession.getActiveSession
相关产品推荐
相关产品推荐
