在Databricks Workflows运行dbt时遇CalledProcessError问题求助

API 探路者

2026-6-13

问题

尝试在Databricks Workflows中构建并调度源自Azure DevOps的dbt项目时，运行dbt出现以下错误：

CalledProcessError: Command 'b'\nmkdir -p &quot;/tmp/tmp-dbt-run-1124228490001263&quot;\nunexpected_errors=&quot;$(cp -a -u &quot;/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/.&quot; &quot;/tmp/tmp-dbt-run-1124228490001263&quot; 2&gt; &gt;(grep -v \'Operation not supported\'))&quot;\nif [[ -n &quot;$unexpected_errors&quot; ]]; then\n  &gt;&amp;2 echo -e &quot;Unexpected error(s) encountered while copying:\n$unexpected_errors&quot;\n  exit 1\nfi\n        returned non-zero exit status 1.

Unexpected error(s) encountered while copying:
cp: cannot stat '/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/./venv/share/doc/networkx-3.1/examples/3d_drawing/__pycache__': No such file or directory
cp: cannot stat '/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/./venv/share/doc/networkx-3.1/examples/algorithms/__pycache__': No such file or directory
cp: cannot stat '/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/./venv/share/doc/networkx-3.1/examples/basic/__pycache__': No such file or directory
cp: cannot stat '/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/./venv/share/doc/networkx-3.1/examples/drawing/__pycache__': No such file or directory
cp: cannot stat '/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/./venv/share/doc/networkx-3.1/examples/graph/__pycache__': No such file or directory
cp: cannot stat '/Workspace/Repos/.internal/085c4ffe5e_commits/16113d05ffd8cd7b148ed973080aa51439e98b0c/./venv/share/doc/networkx-3.1/examples/subclass/__pycache__': No such file or directory

推测问题出在仓库文件复制环节，任务配置如下：

resources:
  jobs:
    otd:
      name: otd
      email_notifications:
        on_failure:
          - mauricio.schwartsman@xxxxxxxx.com
        no_alert_for_skipped_runs: true
      notification_settings:
        no_alert_for_skipped_runs: true
        no_alert_for_canceled_runs: true
      tasks:
        - task_key: otd_dbt
          dbt_task:
            project_directory: &quot;&quot;
            commands:
              - dbt deps
              - dbt build -s +otd_total
            schema: gold
            warehouse_id: xxxxxxxxxxx
            catalog: logistics_prd
            source: GIT
          job_cluster_key: dbt_CLI
          libraries:
            - pypi:
                package: dbt-databricks&gt;=1.0.0,&lt;2.0.0
      job_clusters:
        - job_cluster_key: dbt_CLI
          new_cluster:
            cluster_name: &quot;&quot;
            spark_version: 15.4.x-scala2.12
            spark_conf:
              spark.master: local[*, 4]
              spark.databricks.cluster.profile: singleNode
            azure_attributes:
              first_on_demand: 1
              availability: ON_DEMAND_AZURE
              spot_bid_max_price: -1
            node_type_id: Standard_D4ds_v5
            custom_tags:
              ResourceClass: SingleNode
            spark_env_vars:
              PYSPARK_PYTHON: /databricks/python3/bin/python3
            enable_elastic_disk: true
            data_security_mode: SINGLE_USER
            runtime_engine: PHOTON
            num_workers: 0
      git_source:
        git_url: https://dev.azure.com/copa-energia/Logistics/_git/dbt_logistica
        git_provider: azureDevOpsServices
        git_branch: main
      queue:
        enabled: true

解决方案

排除虚拟环境与缓存文件：错误显示复制时找不到venv下的__pycache__文件，说明本地虚拟环境目录被提交到Git仓库。在项目根目录添加.gitignore文件，写入以下内容：
```
venv/
__pycache__/
*.pyc
```
将.gitignore提交到Azure DevOps仓库，避免无关目录被同步到Databricks。
指定dbt项目目录：当前配置中project_directory为空，导致Databricks尝试复制整个仓库内容。若dbt项目在仓库根目录，设置project_directory: "."；若在子目录，填写对应路径（如dbt_project/），减少不必要的复制操作。
清理已提交的冗余文件：手动删除Azure DevOps仓库中已提交的venv和__pycache__目录，执行以下Git命令后提交：
```
git rm -r --cached venv
git commit -m "Remove venv directory from repo"
git push
```

自定义复制逻辑（临时方案）：若无法立即清理仓库，可改用自定义脚本任务替代默认dbt任务，在复制命令中排除venv目录并扩展错误过滤：

mkdir -p "/tmp/tmp-dbt-run"
unexpected_errors="$(cp -a -u --exclude='venv' "/Workspace/Repos/.internal/your-commit-path/." "/tmp/tmp-dbt-run" 2> >(grep -v 'Operation not supported' | grep -v 'cannot stat'))"
if [[ -n "$unexpected_errors" ]]; then
  >&2 echo -e "Unexpected error(s) encountered while copying:\n$unexpected_errors"
  exit 1
fi
cd /tmp/tmp-dbt-run
dbt deps
dbt build -s +otd_total

内容的提问来源于stack exchange，提问作者Maurício Schwartsman