提交一次性临时Databricks Notebook任务时如何指定集群池？

Got it, let's work through this! You're right that the jobs/runs/submit API doesn't have a top-level instance_pool_id parameter when using an existing cluster. But here's the straightforward workaround: instead of targeting a pre-created cluster with existing_cluster_id, you can define a temporary cluster via the new_cluster field and link it directly to your instance pool. Databricks will spin up a cluster from the pool specifically for your one-time job, then automatically tear it down once the task finishes—exactly what you need for this use case.

Here's your updated PowerShell code to implement this:

$json = @"
{
  "run_name": "integration testing notebook task",
  "timeout_seconds": 3600,
  "new_cluster": {
    "instance_pool_id": "$global:instancePoolID",
    # Optional: Only include these if your instance pool doesn't already set them
    "spark_version": "13.3.x-scala2.12",
    "node_type_id": "Standard_DS3_v2",
    "num_workers": 2
  },
  "notebook_task": {
    "notebook_path": "$global:notebookPath"
  }
}
"@

# Example submission command (adjust the URI and auth token to match your workspace)
Invoke-RestMethod -Uri "https://<your-workspace-url>.azuredatabricks.net/api/2.0/jobs/runs/submit" `
  -Method Post `
  -Headers @{"Authorization" = "Bearer $global:databricksToken"} `
  -Body $json `
  -ContentType "application/json"

Quick Key Details: #

• Replace $global:instancePoolID with your actual instance pool ID (you can find this in the Instance Pools section of your Databricks workspace).
• If your instance pool already has preset configurations (like Spark version, node type, or worker count), you can skip those fields in new_cluster—Databricks will inherit all settings directly from the pool.
• The temporary cluster spun up from the pool will be automatically deleted right after your job completes, so no manual cleanup is needed.
• Double-check that your user or service principal has permissions to access the instance pool and provision clusters from it (verify workspace IAM and instance pool-level permissions if you hit access errors).

内容的提问来源于stack exchange，提问作者user13800089