如何在GitHub Actions单次推送中运行多个Databricks笔记本？

原配置的核心问题是databricks/run-notebook@v0 Action的local-notebook-path参数仅支持单个文件路径，无法直接批量处理多个Notebook。以下是两种可行的修改方案：

方案一：使用矩阵作业（推荐） #

将检测修改文件和执行Notebook拆分为两个独立作业，通过矩阵遍历每个Notebook，实现独立执行：

name: run-sql-notebooks
 
on:
  push:
    branches:
      - dev
 
env:
  DATABRICKS_HOST: https://mydatabricks.cloud.databricks.com
  DATABRICKS_CLUSTER_ID: 1255-122012-pwryxyz1
  DEV: dev
 
jobs:
  detect_changed_notebooks:
    name: 检测修改的SQL Notebook
    runs-on: ubuntu-latest
    outputs:
      notebooks: ${{ steps.changed_sqls.outputs.notebooks }}
    steps:
      - name: 拉取代码仓库
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
 
      - name: 获取最后一次提交中修改的SQL Notebook
        id: changed_sqls
        run: |
          # 提取修改的SQL文件
          NOTEBOOKS=$(git diff --name-only HEAD^ HEAD | grep '\.sql$' || true)
          # 转换为JSON数组格式供矩阵解析
          if [ -z "$NOTEBOOKS" ]; then
            echo "notebooks=[]" >> $GITHUB_OUTPUT
          else
            NOTEBOOKS_JSON=$(echo "$NOTEBOOKS" | jq -R . | jq -s .)
            echo "notebooks=$NOTEBOOKS_JSON" >> $GITHUB_OUTPUT
          fi
 
  run_sql_notebooks:
    name: 执行Notebook - ${{ matrix.notebook }}
    runs-on: ubuntu-latest
    needs: detect_changed_notebooks
    if: needs.detect_changed_notebooks.outputs.notebooks != '[]'
    strategy:
      matrix:
        notebook: ${{ fromJson(needs.detect_changed_notebooks.outputs.notebooks) }}
    steps:
      - name: 拉取代码仓库
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
 
      - name: 在Databricks上执行SQL Notebook
        uses: databricks/run-notebook@v0
        with:
          run-name: "GitHub Actions - ${{ github.run_number }} - ${{ matrix.notebook }}"
          local-notebook-path: ${{ matrix.notebook }}
          git-commit: ${{ github.sha }}
          existing-cluster-id: ${{ env.DATABRICKS_CLUSTER_ID }}
          notebook-params-json: |
            {
            "env": "${{ env.DEV }}"
            }
        env:
          DATABRICKS_HOST: ${{ env.DATABRICKS_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.DB_SECRET }}

关键修改点 #

• 新增detect_changed_notebooks前置作业，专门负责检测修改的Notebook并输出JSON数组格式的列表
• 使用jq工具将换行分隔的文件列表转为GitHub矩阵能识别的JSON数组
• 主作业通过needs依赖前置结果，利用strategy.matrix遍历每个Notebook，每个文件对应独立的执行任务，失败状态单独可见

方案二：使用Bash循环单步骤执行 #

如果不需要独立任务，也可以在单个步骤里循环遍历文件列表，通过Databricks CLI批量执行：

name: run-sql-notebooks
 
on:
  push:
    branches:
      - dev
 
env:
  DATABRICKS_HOST: https://mydatabricks.cloud.databricks.com
  DATABRICKS_CLUSTER_ID: 1255-122012-pwryxyz1
  DEV: dev
 
jobs:
  run_sql_notebooks:
    name: 执行更新的SQL Notebook
    runs-on: ubuntu-latest
 
    steps:
      - name: 拉取代码仓库
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
 
      - name: 获取最后一次提交中修改的SQL Notebook
        id: changed_sqls
        run: |
          NOTEBOOKS=$(git diff --name-only HEAD^ HEAD | grep '\.sql$' || true)
          echo "notebooks<<EOF" >> $GITHUB_OUTPUT
          echo "$NOTEBOOKS" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT
 
      - name: 在Databricks上批量执行SQL Notebook
        if: steps.changed_sqls.outputs.notebooks != ''
        run: |
          # 安装Databricks CLI
          pip install databricks-cli
          # 配置Databricks CLI
          databricks configure --token --host $DATABRICKS_HOST <<EOF
          $DATABRICKS_TOKEN
          EOF
          # 遍历每个Notebook执行
          while IFS= read -r notebook; do
            if [ -n "$notebook" ]; then
              echo "开始执行Notebook: $notebook"
              databricks jobs run-now \
                --notebook-path "/Repos/${{ github.repository }}/$notebook" \
                --existing-cluster-id $DATABRICKS_CLUSTER_ID \
                --notebook-params '{"env": "'"$DEV"'"}'
            fi
          done <<< "${{ steps.changed_sqls.outputs.notebooks }}"
        env:
          DATABRICKS_HOST: ${{ env.DATABRICKS_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.DB_SECRET }}

注意事项 #

• 需要提前确保Notebook在Databricks Repos中的路径与仓库路径一致
• 单个步骤中执行所有Notebook，一个失败会导致后续任务终止，日志较为混杂

内容的提问来源于stack exchange，提问作者Anu