配置带HDFS存储的MLflow追踪服务器时遇hadoop文件未找到错误

我正在搭建带有HDFS artifact store的MLflow追踪服务器，但在Jupyter Notebook中尝试将模型记录到HDFS时，收到如下错误：

File /usr/lib/python3.10/subprocess.py:1845, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session)
   1843     if errno_num != 0:
   1844         err_msg = os.strerror(errno_num)
-> 1845     raise child_exception_type(errno_num, err_msg, err_filename)
   1846 raise child_exception_type(err_msg)

FileNotFoundError: [Errno 2] No such file or directory: 'hadoop'

我已安装并配置了Hadoop，认为相关环境变量设置正确，如下所示：

export PATH="/home/auto/miniconda/bin:$PATH"

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export HADOOP_HOME=/home/auto/hadoop-3.3.4

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

补充：Jupyter使用的环境变量如下：

SHELL=/bin/bash
SESSION_MANAGER=local/auto-virtual-machine:@/tmp/.ICE-unix/1684,unix/auto-virtual-machine:/tmp/.ICE-unix/1684
QT_ACCESSIBILITY=1
COLORTERM=truecolor
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
SSH_AGENT_LAUNCHER=gnome-keyring
XDG_MENU_PREFIX=gnome-
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GNOME_SHELL_SESSION_MODE=ubuntu
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
XMODIFIERS=@im=ibus
DESKTOP_SESSION=ubuntu
GTK_MODULES=gail:atk-bridge
DBUS_STARTER_BUS_TYPE=session
PWD=/home/auto/test
LOGNAME=auto
XDG_SESSION_DESKTOP=ubuntu
XDG_SESSION_TYPE=wayland
JPY_SESSION_NAME=MLflow Live Demo.ipynb
SYSTEMD_EXEC_PID=1684
_=/usr/bin/env
XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.6CXZW1
HOME=/home/auto
USERNAME=auto
IM_CONFIG_PHASE=1
LANG=en_US.UTF-8
XDG_CURRENT_DESKTOP=ubuntu:GNOME
VTE_VERSION=6800
WAYLAND_DISPLAY=wayland-0
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/534f59fb_c9a6_456a_a38d_0f37b80cf726
CLICOLOR=1
GNOME_SETUP_DISPLAY=:1
JPY_PARENT_PID=15426
KMP_DUPLICATE_LIB_OK=True
KMP_INIT_AT_FORK=FALSE
XDG_SESSION_CLASS=user
TERM=xterm-color
USER=auto
GIT_PAGER=cat
GNOME_TERMINAL_SERVICE=:1.188
DISPLAY=:0
SHLVL=0
PAGER=cat
QT_IM_MODULE=ibus
DBUS_STARTER_ADDRESS=unix:path=/run/user/1000/bus,guid=abe24557a2edb707c37a5e24639b3f1d
MPLBACKEND=module://matplotlib_inline.backend_inline
XDG_RUNTIME_DIR=/run/user/1000
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
GDMSESSION=ubuntu
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus,guid=abe24557a2edb707c37a5e24639b3f1d

问题原因及解决方法 #

原因 #

对比你设置的Hadoop环境变量和Jupyter实际使用的环境变量，能看到Jupyter的PATH里没有包含$HADOOP_HOME/bin和$HADOOP_HOME/sbin路径，导致系统找不到hadoop命令。这一般是因为你在终端中设置的环境变量仅对当前终端会话生效，而Jupyter是通过桌面环境启动的，没有继承这些变量。

解决方法 #

• 持久化环境变量：把Hadoop的环境变量配置写入~/.bashrc或~/.profile文件中，执行source ~/.bashrc（bash终端）后重新启动Jupyter Notebook，或者直接重启系统。
• 在Jupyter中临时设置：在Notebook代码开头添加以下代码，手动注入Hadoop相关环境变量：

  import os
  os.environ['HADOOP_HOME'] = '/home/auto/hadoop-3.3.4'
  os.environ['PATH'] += ':/home/auto/hadoop-3.3.4/bin:/home/auto/hadoop-3.3.4/sbin'
  

• 从终端启动Jupyter：在已经配置好Hadoop环境变量的终端中执行jupyter notebook命令，让Jupyter继承终端的环境变量。

内容的提问来源于stack exchange，提问作者Mohamed Naji