Hadoop 2.10.2自定义tmp目录配置失败启动报错求助

问题背景 #

• 使用Hadoop 2.10.2版本，启动NodeManager时失败
• 根因：公司安全策略限制/tmp目录无执行权限，导致无法加载库文件libleveldbjni-64-1-7115603708108814032.8

已尝试操作 #

• 在/opt/osa/hadoop-2.10.2/etc/hadoop/core-site.xml中添加自定义临时目录配置：

<property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/osa/hadoop-2.10.2/tmp_Hadoop</value> 
    <description>A base for other temporary directories.</description> 
</property>

• 重启Hadoop服务并格式化NameNode后，自定义tmp_Hadoop目录仍为空，问题未解决

报错日志 #

2024-06-13 09:57:00,871 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-7115603708108814032.8: /tmp/libleveldbjni-64-1-7115603708108814032.8: failed to map segment from shared object]
at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:1699)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1687)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:355)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:281)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:358)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
2024-06-13 09:57:00,905 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state STOPPED; cause: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:482)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:222)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:104)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67)
at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:482)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:222)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:104)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:67)
at org.apache.hadoop.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:184)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2024-06-13 10:33:36,622 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

解决方案 #

问题本质 #

hadoop.tmp.dir仅控制Hadoop自身的临时文件，但leveldbjni加载库时使用的是JVM默认的/tmp目录，跟Hadoop配置无关，所以修改core-site.xml无法解决问题。

1. 给NodeManager指定JVM临时目录 #

编辑/opt/osa/hadoop-2.10.2/etc/hadoop/yarn-env.sh，添加以下配置：

export YARN_NODEMANAGER_OPTS="-Djava.io.tmpdir=/opt/osa/hadoop-2.10.2/tmp_Hadoop $YARN_NODEMANAGER_OPTS"

然后给自定义目录设置正确权限（假设Hadoop运行用户为hadoop）：

chown -R hadoop:hadoop /opt/osa/hadoop-2.10.2/tmp_Hadoop
chmod 755 /opt/osa/hadoop-2.10.2/tmp_Hadoop

2. 重启验证 #

重启NodeManager服务，检查tmp_Hadoop目录下是否生成leveldb相关库文件，同时查看日志确认加载失败的报错消失。

3. 全局生效（可选） #

如果其他Hadoop服务也遇到类似问题，编辑/opt/osa/hadoop-2.10.2/etc/hadoop/hadoop-env.sh添加全局JVM参数：

export HADOOP_OPTS="-Djava.io.tmpdir=/opt/osa/hadoop-2.10.2/tmp_Hadoop $HADOOP_OPTS"

内容的提问来源于stack exchange，提问作者Prakash