如何通过PowerShell或Databricks API将初始化脚本从DBFS迁移至工作区？

核心格式说明 #

工作区路径的初始化脚本需使用workspace:前缀（Databricks PowerShell模块参数），或在API请求中通过workspace对象指定完整工作区路径，格式示例：workspace:/Workspace/Shared/Init/pyodbc.sh

方法一：使用Databricks PowerShell模块 #

1. 上传脚本至工作区 #

先将本地初始化脚本上传到Databricks工作区指定目录：

# 上传本地pyodbc.sh到工作区/Workspace/Shared/Init路径
New-DatabricksFile -BearerToken $ADB_Token -Region $region `
  -LocalPath "C:\local-scripts\pyodbc.sh" `
  -Path "/Workspace/Shared/Init/pyodbc.sh" `
  -Overwrite

2. 修改集群部署命令 #

更新New-DatabricksCluster的InitScripts参数为工作区路径：

New-DatabricksCluster -BearerToken $ADB_Token -Region $region -ClusterName $cname -SparkVersion $csparkV `
-NodeType $cnodeT -MinNumberOfWorkers $cminWorker -MaxNumberOfWorkers $cmaxWorker -AutoTerminationMinutes $cterm `
-InitScripts "workspace:/Workspace/Shared/Init/pyodbc.sh" -UniqueNames -Update

方法二：直接调用Databricks REST API #

1. 上传脚本至工作区 #

通过workspace/import API上传本地脚本：

$apiBaseUrl = "https://$region.azuredatabricks.net/api/2.0"
$authHeader = @{ "Authorization" = "Bearer $ADB_Token" }

# 读取本地脚本并转为Base64编码
$scriptContent = [Convert]::ToBase64String([IO.File]::ReadAllBytes("C:\local-scripts\pyodbc.sh"))

$uploadBody = @{
  path = "/Workspace/Shared/Init/pyodbc.sh"
  format = "AUTO"
  language = "SH"
  content = $scriptContent
  overwrite = $true
} | ConvertTo-Json

Invoke-RestMethod -Uri "$apiBaseUrl/workspace/import" -Headers $authHeader -Method Post -Body $uploadBody -ContentType "application/json"

2. 创建/更新集群 #

通过clusters/create或clusters/edit API指定工作区路径的初始化脚本：

# 创建集群示例（更新集群用clusters/edit API，需传入cluster_id）
$clusterBody = @{
  cluster_name = $cname
  spark_version = $csparkV
  node_type_id = $cnodeT
  autoscale = @{ min_workers = $cminWorker; max_workers = $cmaxWorker }
  auto_termination_minutes = $cterm
  init_scripts = @(
    @{
      workspace = @{ path = "/Workspace/Shared/Init/pyodbc.sh" }
    }
  )
  unique_name = $true
} | ConvertTo-Json

Invoke-RestMethod -Uri "$apiBaseUrl/clusters/create" -Headers $authHeader -Method Post -Body $clusterBody -ContentType "application/json"

注意事项 #

• 确保操作账号/服务主体拥有工作区文件的读写权限，以及集群创建/编辑权限
• 工作区路径需与实际上传位置完全匹配，区分大小写
• 若使用现有集群更新，PowerShell中保留-Update参数，API需调用clusters/edit接口并传入目标集群ID

内容的提问来源于stack exchange，提问作者Greencolor