使用REST API将自定义Python WHL包部署到Azure Synapse工作区并关联Spark池的问题

我来帮你梳理下正确的实现步骤，你之前直接用PATCH更新Spark池没成功，主要是跳过了上传包到存储和在工作区注册包这两个关键前置环节，下面一步步来：

第一步：把WHL包上传到Synapse关联的ADLS Gen2存储 #

Synapse工作区默认会关联一个ADLS Gen2存储账户，所有自定义库都存在这个存储的synapsefilesys容器下的synapse/workspaces/libraries/路径里。你需要先把本地的WHL文件上传到这个位置，用存储的REST API实现：

Python requests代码示例（用Azure AD认证） #

import requests

# 替换为你的存储账户名和包名
storage_account = "your-storage-account"
package_name = "your-custom-package.whl"
local_package_path = "./path/to/your/package.whl"

# 替换为你的Azure AD访问令牌（可以用msal库获取，后面会讲）
access_token = "your-storage-access-token"

# 构建上传请求地址
upload_url = f"https://{storage_account}.dfs.core.windows.net/synapsefilesys/synapse/workspaces/libraries/{package_name}"

# 读取本地包文件
with open(local_package_path, "rb") as f:
    package_data = f.read()

# 发送PUT请求上传文件
headers = {
    "Authorization": f"Bearer {access_token}",
    "x-ms-blob-type": "BlockBlob",
    "x-ms-version": "2021-06-08"
}

response = requests.put(upload_url, data=package_data, headers=headers)
response.raise_for_status()  # 检查是否上传成功
print("包上传到存储成功！")

第二步：在Synapse工作区注册这个包（推荐，便于管理） #

上传完文件后，需要调用Synapse的管理API把这个包注册到工作区，这样后续关联Spark池时更规范：

Python requests代码示例 #

import requests
import json

# 替换为你的Azure资源信息
subscription_id = "your-sub-id"
resource_group = "your-resource-group"
workspace_name = "your-synapse-workspace"
package_name = "your-custom-package.whl"
storage_account = "your-storage-account"

# 替换为Synapse管理API的访问令牌（同样用msal获取，资源是https://management.azure.com/）
access_token = "your-management-access-token"

# 构建注册请求地址
register_url = f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Synapse/workspaces/{workspace_name}/workspacePackages/{package_name}?api-version=2021-06-01"

# 注册请求体
payload = {
    "properties": {
        "packageUrl": f"abfss://synapsefilesys@{storage_account}.dfs.core.windows.net/synapse/workspaces/libraries/{package_name}",
        "type": "Whl"
    }
}

headers = {
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/json"
}

response = requests.put(register_url, data=json.dumps(payload), headers=headers)
response.raise_for_status()
print("包在Synapse工作区注册成功！")

第三步：关联WHL包到Synapse Spark池 #

现在你可以用PATCH API更新Spark池的自定义库了，注意不要直接覆盖原有库，最好先GET当前的自定义库列表，再把新包加进去：

Python requests代码示例 #

import requests
import json

# 替换为你的Spark池信息
big_data_pool_name = "your-spark-pool"

# 1. 先获取当前Spark池的自定义库列表
get_pool_url = f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Synapse/workspaces/{workspace_name}/bigDataPools/{big_data_pool_name}?api-version=2021-06-01"
headers = {
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/json"
}

response = requests.get(get_pool_url, headers=headers)
response.raise_for_status()
current_pool_config = response.json()
current_custom_libs = current_pool_config.get("properties", {}).get("customLibraries", [])

# 2. 添加新包到列表（避免覆盖原有库）
new_lib = {
    "name": package_name,
    "path": f"abfss://synapsefilesys@{storage_account}.dfs.core.windows.net/synapse/workspaces/libraries/{package_name}",
    "type": "Whl"
}
current_custom_libs.append(new_lib)

# 3. 发送PATCH请求更新Spark池
patch_payload = {
    "properties": {
        "customLibraries": current_custom_libs
    }
}

patch_url = get_pool_url  # 和GET地址一致
response = requests.patch(patch_url, data=json.dumps(patch_payload), headers=headers)
response.raise_for_status()
print("包关联到Spark池成功！")

补充：在GitLab CI/CD中获取Azure AD令牌 #

你可以用msal库通过服务主体获取令牌，在GitLab CI里把服务主体的租户ID、客户端ID、客户端密钥设为CI/CD变量：

import msal

# 从GitLab CI变量中读取服务主体信息
tenant_id = "${AZURE_TENANT_ID}"
client_id = "${AZURE_CLIENT_ID}"
client_secret = "${AZURE_CLIENT_SECRET}"

# 获取管理API的令牌（用于Synapse操作）
management_resource = "https://management.azure.com/"
app = msal.ConfidentialClientApplication(
    client_id, client_secret,
    authority=f"https://login.microsoftonline.com/{tenant_id}"
)
result = app.acquire_token_for_client(scopes=[f"{management_resource}/.default"])
management_token = result["access_token"]

# 获取存储API的令牌（用于上传文件）
storage_resource = "https://storage.azure.com/"
result = app.acquire_token_for_client(scopes=[f"{storage_resource}/.default"])
storage_token = result["access_token"]

你之前的问题分析 #

你直接用PATCH更新Spark池时，路径用了相对路径synapse/workspaces/libraries/{library_name}，但需要用完整的abfss协议路径；另外，包还没上传到存储并注册，所以Spark池找不到对应的文件。按照上面的步骤就能解决啦！

备注：内容来源于stack exchange，提问作者Worldmaster