Databricks Merge操作触发LOCATION_OVERLAP错误求助

场景说明 #

通过调度Notebook每日从PostgreSQL读取数据，使用Merge操作实现Delta表的去重持久化，核心代码如下：

delta_table = DeltaTable.forPath(spark, delta_table_path)

(delta_table
    .alias("target")
    .merge(
        remote_df.alias("source"),
        "target.time = source.time AND target.devicename = source.devicename")  # 定义唯一匹配键
    .whenMatchedUpdateAll()  # 匹配时全量更新
    .whenNotMatchedInsertAll()  # 无匹配时插入新行
    .execute()
)   

Catalog/Schema创建及报错情况 #

预先在S3中创建外部存储位置 s3://databricks-managed-location，执行以下SQL创建Catalog和Schema：

create_catalog_sql = f"CREATE CATALOG IF NOT EXISTS catalog123 MANAGED LOCATION 's3://databricks-managed-location/catalog123';" 
create_schema_sql = f"CREATE SCHEMA IF NOT EXISTS catalog123.myschema MANAGED LOCATION 's3://databricks-managed-location/catalog123';"

执行表初始化语句：

remote_df.write.mode("overwrite").saveAsTable(f"catalog123.myschema.mytable")

触发错误：

Input path url 's3://databricks-managed-location/project123' overlaps with managed storage within 'CheckPathAccess' call. .

更新尝试1 #

修改Schema创建语句，移除MANAGED LOCATION指定：

create_catalog_sql = f"CREATE CATALOG IF NOT EXISTS catalog123 MANAGED LOCATION 's3://databricks-managed-location/catalog123';" 
create_schema_sql = f"CREATE SCHEMA IF NOT EXISTS catalog123.myschema"

错误依旧存在：

's3://databricks-managed-location/catalog123' overlaps with managed storage within 'CheckPathAccess' call. .

更新尝试2 #

调整Schema创建语句，不指定Catalog前缀：

create_catalog_sql = f"CREATE CATALOG IF NOT EXISTS catalog123 MANAGED LOCATION 's3://databricks-managed-location/catalog123';" 
create_schema_sql = f"CREATE SCHEMA IF NOT EXISTS myschema"

该语句执行成功，但myschema未归属到catalog123下，而是默认存储在hive_metastore目录中，不符合预期。

内容的提问来源于stack exchange，提问作者binaryguy