迁移dbx至Databricks CLI:按目标配置Wheel作业参数文件问题
从dbx迁移至Databricks CLI的Bundle配置问题
从dbx迁移到Databricks CLI过程中,我在官方文档及databricks-cli仓库里都没找到对应解决方案。官方的迁移示例过于简略,完全没覆盖作业参数文件这类部署细节。
我的需求是通过Bundle部署带文件参数的Python Wheel作业,当前配置如下:
# The main job for package_name artifacts: package_wheel: build: poetry build path: .. type: whl config_file: build: echo Under build files: - source: test_conf.yaml path: ../conf type: yaml resources: jobs: package_name_job: name: package_name_job schedule: quartz_cron_expression: '44 37 8 * * ?' timezone_id: Europe/Amsterdam email_notifications: on_failure: - my.name@mycompany.com tasks: - task_key: main_task job_cluster_key: job_cluster python_wheel_task: package_name: package_name entry_point: main parameters: ["--conf-file", config_file] libraries: # By default we just include the .whl file generated for the package_name package. - whl: ../dist/*.whl job_clusters: - job_cluster_key: job_cluster new_cluster: spark_version: 13.3.x-scala2.12 node_type_id: n1-standard-4 autoscale: min_workers: 1 max_workers: 4
我希望根据databricks.yaml中定义的targets,为作业部署不同的config_file,但无法让databricks-cli将这些文件识别为artifacts并上传至.bundle/[package_name]/[target]/files路径(Wheel包可正常上传至.bundle/[package_name]/[target]/artifacts)。
我尝试将config_file定义为artifact并引用,但未成功:
# The main job for package_name artifacts: ... config_file: build: echo Under build files: - source: test_conf.yaml path: ../conf type: yaml resources: jobs: package_name_job: name: package_name_job ... tasks: - task_key: main_task job_cluster_key: job_cluster python_wheel_task: package_name: package_name entry_point: main parameters: ["--conf-file", ${artifacts.config_file}] # <-- Reference as in terraform? ...
内容的提问来源于stack exchange,提问作者nenetto
相关产品推荐
相关产品推荐

