使用dbutils.notebook.run()仅显示通用WorkflowException，无法获取具体错误

问题场景 #

在Databricks的Python文件中使用dbutils.notebook.run()执行子Notebook时，子Notebook失败仅返回通用错误日志，无法获取直接运行子Notebook时的真实具体错误信息，需要优化以捕获具体错误。

现有代码 #

absolute_notebook_path = '/Workspace/company/ETL-Processing/packages/example/src/scripts/merge_to_gold.py'
try:
    dbutils.notebook.run(absolute_notebook_path, 3600)
    print(f"Successfully ran notebook: {absolute_notebook_path}")
except Exception as e:
    error_message = str(e)
    raise Exception(f"Notebook '{absolute_notebook_path}' failed with error: {error_message}")

通用错误日志 #

Error: An error occurred while calling o560.run. : com.databricks.WorkflowException:
com.databricks.NotebookExecutionException: 
FAILED: Workload failed, see run output for details at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:99) 
at com.databricks.dbutils_v1.impl.NotebookUtilsImpl.run(NotebookUtilsImpl.scala:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) 
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:306) 
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) 
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) 
at java.lang.Thread.run(Thread.java:750) 
Caused by: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details at com.databricks.workflow.WorkflowDriver.run0(WorkflowDriver.scala:147) 
at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:94) ... 13 more

真实具体错误日志 #

FileNotFoundError: [Errno 2] No such file or directory:
'/Workspace/company/ETL-processing/packages/membership/src/tables/silver/memberships/table.py'

解决方案 #

方法1：通过子Notebook返回错误信息 #

修改子Notebook，在捕获异常时通过dbutils.notebook.exit()传递具体错误内容：

# 子Notebook代码
try:
    # 业务逻辑代码
    pass
except Exception as e:
    # 将错误信息以特定标识开头返回
    dbutils.notebook.exit(f"ERROR::{str(e)}")

然后在父Notebook中接收返回值并判断是否为错误：

absolute_notebook_path = '/Workspace/company/ETL-Processing/packages/example/src/scripts/merge_to_gold.py'
try:
    run_result = dbutils.notebook.run(absolute_notebook_path, 3600)
    # 检查返回结果是否包含错误标识
    if run_result.startswith("ERROR::"):
        specific_error = run_result[7:]
        raise Exception(specific_error)
    print(f"Successfully ran notebook: {absolute_notebook_path}")
except Exception as e:
    raise Exception(f"Notebook '{absolute_notebook_path}' failed with error: {str(e)}")

方法2：使用%run魔法命令（调试阶段适用） #

如果是在交互式Notebook中执行，而非独立Python脚本，直接使用%run命令，它会直接展示子Notebook的具体错误：

%run "/Workspace/company/ETL-Processing/packages/example/src/scripts/merge_to_gold.py"

注意：%run为同步执行，无法设置超时时间，适合开发调试场景。

方法3：通过工作流API获取详细日志 #

如果是在生产环境的工作流中执行，可以调用Databricks Jobs API获取子Notebook的执行日志详情，解析后提取具体错误。这种方式适合自动化流水线场景，但需要配置API权限。

内容的提问来源于stack exchange，提问作者Itai Sharon