Azure Databricks无服务器SQL仓库连接Collibra遇403权限问题求助

我们正尝试从Collibra连接Azure Databricks Serverless SQL Warehouse，以获取Unity Catalog中的血缘数据。已创建对应的App Registration并为UC授予了适当权限，同时完成了IP地址白名单配置，且通过traceroute验证了连通性正常。但通过curl测试端点时失败，返回403错误：

{"error_code":403,"message":"Invalid access to Org: 5377825XX8869582"}

注：使用通用JDBC连接器连接Hivemetastore可正常工作。

curl测试输出 #

[root@OurEdgeServer1 ~]# curl -v -u token https://adb-9999888877776666.2.azuredatabricks.net/api/2.1/unity-catalog/catalogs
Enter host password for user 'token':
*   Trying 22.42.XX.XX...
* TCP_NODELAY set
* Connected to adb-9999888877776666.2.azuredatabricks.net (22.42.XX.XX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Databricks Inc.; CN=*.azuredatabricks.net
*  start date: Jun  4 00:00:00 2024 GMT
*  expire date: Oct 11 23:59:59 2024 GMT
*  subjectAltName: host "adb-9999888877776666.2.azuredatabricks.net" matched cert's "*.2.azuredatabricks.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Server auth using Basic with user 'token'
* Using Stream ID: 1 (easy handle 0x55f0fa6eb6f0)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /api/2.1/unity-catalog/catalogs HTTP/2
> Host: adb-9999888877776666.2.azuredatabricks.net
> Authorization: Basic 11113333ZGFwaWQ1MTk0ZTZkNTUxMDhiZGQwYjZkYTRhY222288889999TI=
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 403
< content-type: application/json; charset=utf-8
< content-length: 70
< x-databricks-reason-phrase: Invalid access to Org: 5377825XX8809580
< vary: Accept-Encoding
< date: Fri, 07 Jun 2024 23:23:40 GMT
< server: databricks
< 
* Connection #0 to host adb-9999888877776666.2.azuredatabricks.net left intact
{"error_code":403,"message":"Invalid access to Org: 5377825XX8809580"}

问题解答 #

1. 如何查看Serverless SQL Warehouse的连接日志，确认请求是否到达？ #

• 登录Azure Databricks工作区，进入服务器less SQL仓库页面，选中目标仓库
• 切换到日志标签页，这里可以查看所有连接请求的详细记录，包括请求来源IP、请求时间、响应状态码等信息
• 也可以通过Azure Monitor集成Databricks日志，筛选SQL Warehouse的连接事件，实现更灵活的日志查询和告警

2. 该403权限错误如何解决？ #

针对"Invalid access to Org"错误，按以下步骤排查：

• 确认App Registration权限：在Azure AD中给目标App分配Databricks SQL相关权限，以及Unity Catalog的必要权限（如USE CATALOG、SELECT等），并且必须完成管理员同意授权
• 验证Token有效性：确保使用的是App Registration的client secret生成的正确token，token的受众（audience）需指向Databricks工作区URL，或通过Azure AD正确获取针对Databricks的访问token
• 核对组织ID匹配：检查错误中的Org ID是否与你的Databricks工作区所属组织ID一致，确认App Registration所在的Azure AD租户与Databricks工作区的租户完全匹配
• 检查IP白名单配置：确认IP白名单是针对Serverless SQL Warehouse单独配置的（而非仅工作区UI），在仓库的设置页面中验证白名单规则是否包含你的Edge Server IP
• 对比Hivemetastore连接差异：由于Hivemetastore使用JDBC认证，而Unity Catalog通过REST API访问，需确认App是否被授予访问Unity Catalog API的权限，可在Databricks工作区的权限管理中检查App的API访问权限

内容的提问来源于stack exchange，提问作者Gopinath Rajee