Hive执行Insert/Join时MapReduce任务失败问题求助

已部署Hadoop 3.3.6 + Hive 4.0.0，MySQL作为元数据存储。create table、select * from等基础语句正常，但执行insert或select join时，MapReduce任务阶段失败，提示"requires lock = true"且无锁创建。已给HDFS仓库目录设chmod 777权限，问题未解决；MR执行引擎下Yarn无日志输出。

报错日志 #

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
No Stats for default@student1, Columns: name, gpa, age
No Stats for default@student2, Columns: nama, jurusan, hobi
Query ID = hadoop_20240923190426_36da4532-a25c-4d21-b7ec-47f18290f3dd
Total jobs = 1
java.lang.NullPointerException  at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:843)
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:458)
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:475)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initialize(MapredLocalTask.java:136)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:766)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Execution failed with exit status: 1
Obtaining error information

Task failed!
Task ID:
  Stage-4

Logs:

/tmp/hadoop/hive.log
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

配置文件 #

core-site.xml #

<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/tmpdata</value>
</property>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
</property>
<property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hadoop.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.server.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.server.groups</name>
    <value>*</value>
</property>

hdfs-site.xml #

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>

hive-site.xml #

<configuration>
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
    <description>metadata is stored in a MySQL server</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
    <description>MySQL JDBC driver class</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hadoop</value>
    <description>user name for connecting to mysql server </description>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>12345</value>
    <description>password for connecting to mysql server </description>
</property>
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://localhost:9083</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
</configuration>

排查方案 #

• 锁定机制配置检查：Hive 4.0默认启用元数据锁，需确保配置正确。在hive-site.xml中添加/调整：

  <!-- 启用锁机制（需部署ZooKeeper） -->
  <property>
      <name>hive.support.concurrency</name>
      <value>true</value>
  </property>
  <property>
      <name>hive.metastore.locks.enabled</name>
      <value>true</value>
  </property>
  <property>
      <name>hive.zookeeper.quorum</name>
      <value>localhost</value>
  </property>
  

  若未部署ZooKeeper，测试环境可临时关闭锁：

  <property>
      <name>hive.support.concurrency</name>
      <value>false</value>
  </property>
  <property>
      <name>hive.metastore.locks.enabled</name>
      <value>false</value>
  </property>
  

• 解决NullPointerException：该异常通常是Hadoop配置加载异常，检查：

  • 确认HADOOP_HOME环境变量正确设置，Hive的conf目录下需有Hadoop配置文件的软链接或副本
  • 检查hadoop.tmp.dir对应的/home/hadoop/tmpdata目录是否存在，执行chown -R hadoop:hadoop /home/hadoop/tmpdata确保权限归属正确

• 统计信息缺失处理：日志提示表无统计信息，手动生成：

  ANALYZE TABLE student1 COMPUTE STATISTICS;
  ANALYZE TABLE student2 COMPUTE STATISTICS;
  

• MR执行引擎配置：确保Hive使用MR引擎，在hive-site.xml添加：

  <property>
      <name>hive.execution.engine</name>
      <value>mr</value>
  </property>
  

  同时用jps命令确认Yarn的ResourceManager、NodeManager进程正常启动

内容的提问来源于stack exchange，提问作者Dzaki Wicaksono