Dataproc模板JDBC转BigQuery时dataproc_templates_distribution.egg上传失败

使用Dataproc模板的jdbc_to_bigquery.py和bin/start.sh工具从JDBC导入数据到BigQuery，在Ubuntu系统执行指定shell脚本时，出现dataproc_templates_distribution.egg文件上传至GCS失败的错误，具体错误日志如下：

/usr/lib/python3/dist-packages/setuptools/dist.py:508: InformationOnly: Normalizing '0.5.0-beta' to '0.5.0b0'
  self.metadata.version = self._normalize_version(
running bdist_egg
running egg_info
writing google_dataproc_templates.egg-info/PKG-INFO
writing dependency_links to google_dataproc_templates.egg-info/dependency_links.txt
writing requirements to google_dataproc_templates.egg-info/requires.txt
writing top-level names to google_dataproc_templates.egg-info/top_level.txt
reading manifest file 'google_dataproc_templates.egg-info/SOURCES.txt'
writing manifest file 'google_dataproc_templates.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying google_dataproc_templates.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying google_dataproc_templates.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying google_dataproc_templates.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying google_dataproc_templates.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying google_dataproc_templates.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/google_dataproc_templates-0.5.0b0-py3.12.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Will rename output .egg file from dist/google_dataproc_templates-0.5.0b0-py3.12.egg to dist/dataproc_templates_distribution.egg
JOB_TYPE is SERVERLESS, so will submit on serverless Spark
Triggering Spark Submit job
gcloud beta dataproc batches submit pyspark <confidential parameters>
ERROR: (gcloud.beta.dataproc.batches.submit.pyspark) Failed to upload files ['./../dist/dataproc_templates_distribution.egg'] to 'gs://dataproc_test_jdbc_bq/dependencies': [./../dist/dataproc_templates_distribution.egg] not found or not accessible

已知条件 #

• dist/dataproc_templates_distribution.egg已在本地生成
• GCS存储桶中已创建dependencies文件夹，包含repo中的dataproc-templates/python/main.py
• 使用gsutil可正常上传该.egg文件至GCS

解决方案 #

• 修正脚本路径引用
  错误提示中的路径./../dist/dataproc_templates_distribution.egg说明脚本执行时的工作目录与预期不符。打开bin/start.sh，将引用.egg文件的相对路径替换为绝对路径（例如/home/your-user/dataproc-templates/dist/dataproc_templates_distribution.egg），避免工作目录变化导致路径失效。
• 手动上传后跳过自动构建步骤
  先用gsutil cp dist/dataproc_templates_distribution.egg gs://dataproc_test_jdbc_bq/dependencies/手动上传文件，然后修改start.sh，在gcloud beta dataproc batches submit pyspark命令中直接使用GCS上的文件路径gs://dataproc_test_jdbc_bq/dependencies/dataproc_templates_distribution.egg，并注释掉脚本中自动构建和上传.egg的相关代码。
• 固定脚本工作目录
  在start.sh开头添加cd "$(dirname "$0")/.."，确保脚本执行时工作目录切换到Dataproc模板仓库的根目录，这样相对路径就能正确指向生成的.egg文件。
• 检查文件权限
  执行ls -l dist/dataproc_templates_distribution.egg确认文件读权限，若当前用户无读权限，执行chmod +r dist/dataproc_templates_distribution.egg添加权限。

内容的提问来源于stack exchange，提问作者Pachu Martinez