如何在Dataproc集群提交Spark Job时指定自定义job_id？

要设置自定义的job_id，只需在构造job字典时添加reference字段，指定你想要的ID即可，这会覆盖系统默认生成的UUID格式ID：

修改后的代码示例：

from google.cloud import dataproc_v1
job_client = dataproc_v1.JobControllerClient(
      client_options={"api_endpoint": "{}-dataproc.googleapis.com:443".format(region)}
  )
job = {
        "reference": {"job_id": "your-custom-job-id"},  # 配置自定义job ID
        "placement": {"cluster_name": cluster_name},
        "spark_job": {
            "main_class": "de.App",
            "jar_file_uris": [xxxxxx],
            "args": [xxxx],
        },
}
operation = job_client.submit_job_as_operation(request={"project_id": project_id, "region": region, "job": job})
response = operation.result()

注意事项 #

• 自定义job_id需符合Dataproc命名规范：仅允许小写字母、数字、连字符（-）和下划线（_），长度不超过100字符
• 同一项目和区域内的job_id必须唯一，重复提交会触发报错

内容的提问来源于stack exchange，提问作者Abhishek Singh