如何在DevOps流水线中调度Databricks笔记本及问题排查

原脚本核心错误 #

• Notebook路径格式错误：/Users/notebook.py/末尾多了斜杠，且Databricks笔记本路径无需添加.py后缀（实际笔记本为.db格式，只需填写逻辑路径如/Users/你的用户名/notebook1）
• Schedule参数冲突：同时使用"schedule": "@daily"和"cron_schedule": "45 8 * * *"，Databricks API要求schedule是包含quartz_cron_expression和timezone_id的对象，不能混用快捷语法与独立cron字段
• 集群配置不完整：new_cluster缺少num_workers或autoscale配置，API会直接拒绝请求
• 未捕获API响应：curl默认不输出响应内容，无法排查具体错误原因

修正后的脚本 #

- task: Bash@3
  displayName: 'Schedule Databricks Notebook'
  inputs:
    targetType: 'inline'
    script: |
      personalAccessToken='dapie7'  
      databricksUrl='https://adb-13946.6.azuredatabricks.net/api/2.0'
      # 修正路径：移除末尾斜杠与.py后缀
      notebookPath='/Users/你的用户名/notebook1'
      jobName='ScheduledJobName'

      requestUri="$databricksUrl/jobs/create"

      # 修正schedule格式，补全集群配置，移除冲突参数
      body='{
        "name": "'$jobName'",
        "new_cluster": {
          "spark_version": "7.0.x-scala2.12", # 指定完整版本号避免歧义
          "node_type_id": "Standard_DS3_v2",
          "num_workers": 0 # 单节点集群，可按需调整
        },
        "notebook_task": {
          "notebook_path": "'$notebookPath'"
        },
        "schedule": {
          "quartz_cron_expression": "45 8 * * * ?", # 使用Quartz标准Cron表达式
          "timezone_id": "Canada/Eastern"
        },
        "max_retries": 0
      }'

      # Encode PAT to Base64
      patBase64=$(echo -n ":$personalAccessToken" | base64)

      # 添加-v参数查看详细请求/响应，便于排查错误
      curl -X POST -v -H "Authorization: Basic $patBase64" -H "Content-Type: application/json" -d "$body" "$requestUri"

原脚本核心错误 #

• API使用错误：jobs/run-now是触发已存在任务的即时运行，不能用来创建新任务，body2中的name、new_cluster都是无效参数，会导致API报错

方案1：创建两个独立任务（一个定时，一个按需触发） #

- task: Bash@3
  displayName: 'Schedule & Run Databricks Notebooks'
  inputs:
    targetType: 'inline'
    script: | 
      databricksUrl='https://adb-13946046.6.azuredatabricks.net/api/2.0'
      notebookPath='/Users/user/notebook1'
      notebookPath1='/Users/user/notebook2'

      jobName='ScheduledJobName1'
      jobName2='testjob'

      requestUri="$databricksUrl/jobs/create"

      # 创建第一个定时任务的请求体
      body='{
        "name": "'$jobName'",
        "new_cluster": {
          "spark_version": "7.0.x-scala2.12",
          "node_type_id": "Standard_DS3_v2",
          "num_workers": 0
        },
        "notebook_task": {
          "notebook_path": "'$notebookPath'"
        },
        "schedule": {
            "quartz_cron_expression": "45 10 * * * ?",
            "timezone_id": "Canada/Eastern"
        }
      }'

      # 创建第二个无定时任务的请求体
      body2='{
        "name": "'$jobName2'",
        "new_cluster": {
          "spark_version": "7.0.x-scala2.12",
          "node_type_id": "Standard_DS3_v2",
          "num_workers": 0
        },
        "notebook_task": {
          "notebook_path": "'$notebookPath1'"
        }
      }'

      # 创建第二个任务并捕获响应，提取job_id
      createResponse=$(curl -X POST -s -H "Authorization: Bearer $(token)" -H "Content-Type: application/json" -d "$body2" "$requestUri")
      jobId=$(echo $createResponse | grep -o '"job_id":[0-9]*' | cut -d':' -f2)

      # 触发第二个任务的即时运行
      if [ ! -z "$jobId" ]; then
        curl -X POST -H "Authorization: Bearer $(token)" -H "Content-Type: application/json" -d '{"job_id": '$jobId'}' "$databricksUrl/jobs/run-now"
      fi

      # 创建第一个定时任务
      curl -X POST -H "Authorization: Bearer $(token)" -H "Content-Type: application/json" -d "$body" "$requestUri"

方案2：创建单个任务，并行运行两个笔记本（更高效） #

若两个笔记本可共享集群，建议创建包含并行任务组的单任务，减少集群资源开销：

- task: Bash@3
  displayName: 'Schedule Parallel Databricks Notebooks'
  inputs:
    targetType: 'inline'
    script: | 
      databricksUrl='https://adb-13946046.6.azuredatabricks.net/api/2.0'
      notebookPath='/Users/user/notebook1'
      notebookPath1='/Users/user/notebook2'

      jobName='ParallelScheduledJob'

      requestUri="$databricksUrl/jobs/create"

      body='{
        "name": "'$jobName'",
        "new_cluster": {
          "spark_version": "7.0.x-scala2.12",
          "node_type_id": "Standard_DS3_v2",
          "num_workers": 2 # 根据并行任务数调整
        },
        "task": {
          "task_key": "parallel_group",
          "parallel_task": {
            "tasks": [
              {
                "task_key": "notebook1_task",
                "notebook_task": {
                  "notebook_path": "'$notebookPath'"
                }
              },
              {
                "task_key": "notebook2_task",
                "notebook_task": {
                  "notebook_path": "'$notebookPath1'"
                }
              }
            ]
          }
        },
        "schedule": {
            "quartz_cron_expression": "45 10 * * * ?",
            "timezone_id": "Canada/Eastern"
        }
      }'

      curl -X POST -H "Authorization: Bearer $(token)" -H "Content-Type: application/json" -d "$body" "$requestUri"

内容的提问来源于stack exchange，提问作者Vin