Dataproc 2.1添加Hudi可选组件触发枚举标签未知ValueError

使用Dataproc 2.1，在JSON配置中设置如下software_config：

"software_config": {
    "properties": {},
    "optional_components": ["JUPYTER","ZEPPELIN","Hudi"]
}

创建集群时触发错误：

[2023-12-14, 04:08:59 IST] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 609, in execute
    operation = self._create_cluster(hook)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 535, in _create_cluster
    return hook.create_cluster(
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/common/hooks/base_google.py", line 468, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/dataproc.py", line 332, in create_cluster
    result = client.create_cluster(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/dataproc_v1/services/cluster_controller/client.py", line 534, in create_cluster
    request = clusters.CreateClusterRequest(request)
  File "/usr/local/lib/python3.9/site-packages/proto/message.py", line 516, in __init__
    pb_value = marshal.to_proto(pb_type, value)
  File "/usr/local/lib/python3.9/site-packages/proto/marshal/marshal.py", line 211, in to_proto
    pb_value = rule.to_proto(value)
  File "/usr/local/lib/python3.9/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
ValueError: unknown enum label "Hudi"

改为"HUDI"后仍触发同类错误：

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 609, in execute
    operation = self._create_cluster(hook)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 535, in _create_cluster
    return hook.create_cluster(
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/common/hooks/base_google.py", line 468, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/dataproc.py", line 332, in create_cluster
    result = client.create_cluster(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/dataproc_v1/services/cluster_controller/client.py", line 534, in create_cluster
    request = clusters.CreateClusterRequest(request)
  File "/usr/local/lib/python3.9/site-packages/proto/message.py", line 516, in __init__
    pb_value = marshal.to_proto(pb_type, value)
  File "/usr/local/lib/python3.9/site-packages/proto/marshal/marshal.py", line 211, in to_proto
    pb_value = rule.to_proto(value)
  File "/usr/local/lib/python3.9/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
    return self._descriptor(**value)
ValueError: unknown enum label "HUDI"

Dataproc 2.1本身支持Hudi，但你使用的Airflow依赖的google-cloud-dataproc客户端库版本过旧，其枚举定义中尚未加入HUDI选项，导致客户端无法识别该标签并报错。

方案1：升级客户端库版本 #

升级Airflow的Google Cloud相关依赖包，确保google-cloud-dataproc和apache-airflow-providers-google版本支持HUDI枚举：

pip install --upgrade google-cloud-dataproc apache-airflow-providers-google

升级完成后，使用"HUDI"作为可选组件标签即可正常创建集群。

方案2：使用初始化动作安装Hudi（无需升级库） #

若无法升级客户端库，可通过Dataproc初始化动作手动安装Hudi，移除optional_components中的Hudi配置，改用initialization_actions指定安装脚本：

"config": {
    "software_config": {
        "properties": {},
        "optional_components": ["JUPYTER","ZEPPELIN"]
    },
    "initialization_actions": [
        {
            "uri": "gs://dataproc-initialization-actions/hudi/hudi.sh",
            "execution_timeout": "300s"
        }
    ]
}

内容的提问来源于stack exchange，提问作者user19930511