macOS Ventura下安装Hadoop遇getSubject错误求助

在macOS Ventura系统上安装Hadoop多次失败，试过3.4.0和3.3.6两个版本，均未解决。当前使用Java版本为java 23。

执行hdfs namenode -format时出现错误：

退出状态1: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

执行start-all.sh时的错误输出：

Starting namenodes
Starting namenodes on [Sumans-MacBook-Pro.local]
Starting datanodes
localhost: ERROR: Cannot set priority of datanode process 3556
2024-09-23 16:10:29,438 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
    at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
    at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:344)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
    at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:361)
Starting resourcemanagers on []
Starting nodemanagers
localhost: ERROR: Cannot set priority of nodemanager process 3966

当前使用的配置文件如下：

core-site.xml #

<configuration>
  <property>
      <name>hadoop.tmp.dir</name>
      <value>/Users/sumanbhattarai/hdfs/tmp/</value>
  </property>
  <property>
      <name>fs.default.name</name>
      <value>hdfs://127.0.0.1:9000</value>
  </property>
</configuration>

hdfs-site.xml #

<configuration>
  <property>
      <name>dfs.data.dir</name>
      <value>/Users/sumanbhattarai/hdfs/namenode</value>
  </property>
  <property>
      <name>dfs.data.dir</name>
      <value>/Users/sumanbhattarai/hdfs/datanode</value>
  </property>
  <property>
      <name>dfs.replication</name>
      <value>1</value>
  </property>
</configuration>

mapred-site.xml #

<configuration> 
  <property> 
    <name>mapreduce.framework.name</name> 
    <value>yarn</value> 
  </property> 
</configuration>

yarn-site.xml #

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>127.0.0.1</value>
  </property>
  <property>
    <name>yarn.acl.enable</name>
    <value>0</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>   
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>

解决方法 #

1. 降级Java版本

Java 23移除了安全管理器相关API，而Hadoop 3.3.x/3.4.x版本依赖这些API，必须降级到Java 8或Java 11（官方推荐的兼容版本）。

2. 修正hdfs-site.xml配置错误

你的hdfs-site.xml中重复定义了dfs.data.dir属性，导致namenode路径被覆盖，正确配置需区分namenode和datanode目录：

<configuration>
  <property>
      <name>dfs.namenode.name.dir</name>
      <value>/Users/sumanbhattarai/hdfs/namenode</value>
  </property>
  <property>
      <name>dfs.datanode.data.dir</name>
      <value>/Users/sumanbhattarai/hdfs/datanode</value>
  </property>
  <property>
      <name>dfs.replication</name>
      <value>1</value>
  </property>
</configuration>

3. 修复权限问题

macOS下启动Hadoop时出现的Cannot set priority of datanode process错误，可通过以下两种方式解决：

• 执行命令修改目录权限：

sudo chown -R $USER:$USER /Users/sumanbhattarai/hdfs

• 修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh，添加配置：

export HADOOP_OPTS="-Djava.security.manager=allow"

（注：此配置仅在降级Java后生效，Java 23下无法使用）

4. 重新格式化namenode

完成上述修改后，先清理旧数据：

rm -rf /Users/sumanbhattarai/hdfs/tmp/*
rm -rf /Users/sumanbhattarai/hdfs/namenode/*
rm -rf /Users/sumanbhattarai/hdfs/datanode/*

再执行格式化命令：

hdfs namenode -format

5. 使用官方推荐的启动方式

start-all.sh是过时脚本，建议分组件启动：

start-dfs.sh
start-yarn.sh

内容的提问来源于stack exchange，提问作者Suman Bhattarai