使用Kubernetes Executor运行Airflow HiveOperator时遇权限拒绝错误

在使用Kubernetes Executor的Apache Airflow中运行HiveOperator任务时遇到权限错误。

Dockerfile配置 #

已编写Dockerfile并安装apache-airflow-providers-apache-hive==6.4.1等依赖：

FROM apache/airflow:2.8.2
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
RUN umask 0002; \
    mkdir -p /tmp

HiveOperator任务定义 #

hive_select = HiveOperator(
        task_id='hive_select',
        hive_cli_conn_id='hive_conn',
        hql="select * from table LIMIT 10",
        execution_timeout=timedelta(minutes=30)
    )

错误日志 #

执行任务时出现以下权限错误：

[2024-02-29, 09:05:15 UTC] {hive.py:275} INFO - hive -hiveconf airflow.ctx.dag_id=hive_con_test -hiveconf airflow.ctx.task_id=hive_select -hiveconf airflow.ctx.execution_date=2024-02-29T09:02:14.534414+00:00 -hiveconf airflow.ctx.try_number=4 -hiveconf airflow.ctx.dag_run_id=manual__2024-02-29T09:02:14.534414+00:00 -hiveconf airflow.ctx.dag_owner=airflow -hiveconf airflow.ctx.dag_email= -hiveconf mapred.job.name=Airflow HiveOperator task for hive-con-test-hive-select-g7oo5n2s.hive_con_test.hive_select.2024-02-29T09:02:14.534414+00:00 -f /tmp/airflow_hiveop_wg4va81k/tmpwyi3r0ow
[2024-02-29, 09:05:15 UTC] {taskinstance.py:2728} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 439, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 414, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/apache/hive/operators/hive.py", line 172, in execute
    self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/apache/hive/hooks/hive.py", line 276, in run_cli
    sub_process: Any = subprocess.Popen(
                       ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.11/subprocess.py", line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: 'hive'
[2024-02-29, 09:05:15 UTC] {taskinstance.py:1149} INFO - Marking task as FAILED. dag_id=hive_con_test, task_id=hive_select, execution_date=20240229T090214, start_date=20240229T090514, end_date=20240229T090515
[2024-02-29, 09:05:15 UTC] {standard_task_runner.py:107} ERROR - Failed to execute job 214 for task hive_select ([Errno 13] Permission denied: 'hive'; 28)
[2024-02-29, 09:05:15 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 1
[2024-02-29, 09:05:15 UTC] {taskinstance.py:3309} INFO - 0 downstream tasks scheduled from follow-on schedule check

核心错误：PermissionError: [Errno 13] Permission denied: 'hive'

已尝试设置/tmp文件夹权限，但不确定操作是否正确，寻求解决该权限问题的具体建议。

解决建议 #

• 确认Hive CLI是否安装在镜像中
  当前Dockerfile仅安装了Airflow的Hive provider，未包含Hive CLI本身。Airflow的HiveOperator默认调用系统中的hive命令，若镜像中没有安装Hive客户端，会导致命令找不到或触发权限错误（实际是命令不存在，但错误提示可能显示权限拒绝）。需在Dockerfile中添加Hive客户端的安装步骤，比如基于Hadoop/Hive官方镜像构建，或者通过包管理器安装。

• 调整任务执行用户权限
  Airflow默认以airflow用户运行任务，若Hive CLI的执行权限未开放给该用户，会触发权限拒绝。可采取两种方式：

  1. 在Dockerfile中切换到root用户安装Hive客户端，之后切回airflow用户，并确保airflow用户对Hive CLI的执行路径有可执行权限。
  2. 在Kubernetes Executor的Pod配置中，指定任务运行的用户为拥有Hive执行权限的用户（需确保集群策略允许）。

• 修正/tmp目录权限
  当前Dockerfile中创建/tmp的命令权限设置不足，需确保/tmp目录对airflow用户有读写执行权限。可在Dockerfile中添加：

  RUN chown airflow:airflow /tmp && chmod 755 /tmp
  

  若需要更宽松的权限（适合测试环境），也可设置：

  RUN chmod 777 /tmp
  

• 改用HiveServer2连接替代CLI
  HiveOperator支持通过HiveServer2的JDBC/Thrift连接执行HQL，无需依赖本地Hive CLI。修改任务配置，使用hiveserver2_conn_id替代hive_cli_conn_id，并确保连接配置正确指向HiveServer2地址。这种方式避免了本地安装Hive CLI的问题，更适配容器化环境。

内容的提问来源于stack exchange，提问作者han