Spring Batch Hive Tasklet执行报错求助：NoClassDefFoundError

问题情况 #

使用Spring Batch的hdp:hive-tasklet执行Hive脚本（即使简化为show databases;）时，始终抛出NoClassDefFoundError错误，但通过Java代码直接使用JDBC连接HiveServer2执行相同查询却能正常运行。

错误日志 #

2022-09-08 18:24:46.095 ERROR AbstractStep:229 - Encountered an error executing step addPartitionToActiveItemsTableScriptTaskletStep in job floorPriceActiveItemsJob
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/service/HiveServerException
    at org.springframework.data.hadoop.hive.HiveTemplate$1.doInHive(HiveTemplate.java:267) ~[spring-data-hadoop-hive-2.5.0.RELEASE.jar:2.5.0.RELEASE]
    ...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.service.HiveServerException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_202]
    ... 40 more

当前配置 #

Hive Tasklet配置 #

<hdp:hive-tasklet id="addPartitionToActiveItemsTableScriptTasklet"
                  scope="step">
    <hdp:script>
        <![CDATA[
        ALTER TABLE active_items DROP IF EXISTS PARTITION
        (batch_date="#{jobExecutionContext['formattedOperationDate']}");
        ALTER TABLE active_items ADD PARTITION
        (batch_date="#{jobExecutionContext['formattedOperationDate']}")
        LOCATION
        "${hdfs.data.path}/${hdfs.data.resource.active.items}/#{jobExecutionContext['formattedOperationDate']}";
    ]]>
    </hdp:script>
</hdp:hive-tasklet>

spring-hadoop.xml配置 #

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:util="http://www.springframework.org/schema/util" xmlns:context="http://www.springframework.org/schema/context"
    xmlns:c="http://www.springframework.org/schema/c" xmlns:p="http://www.springframework.org/schema/p"
    xmlns:batch="http://www.springframework.org/schema/batch" xmlns:task="http://www.springframework.org/schema/task"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:hdp="http://www.springframework.org/schema/hadoop"
    xsi:schemaLocation="http://www.springframework.org/schema/context
                http://www.springframework.org/schema/context/spring-context.xsd
                http://www.springframework.org/schema/batch
                http://www.springframework.org/schema/batch/spring-batch.xsd
                http://www.springframework.org/schema/beans 
                http://www.springframework.org/schema/beans/spring-beans.xsd
                http://www.springframework.org/schema/util
                http://www.springframework.org/schema/util/spring-util.xsd
                http://www.springframework.org/schema/hadoop
                http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">

    <hdp:configuration id="hdpConfiguration">
        fs.defaultFS=hdfs://localhost:9000
    </hdp:configuration>

    <bean id="hiveDriver" class="org.apache.hive.jdbc.HiveDriver" />
    <bean id="hiveDataSource"
        class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
        <constructor-arg name="driver" ref="hiveDriver" />
        <constructor-arg name="url" value="${hive.url}" />
        <constructor-arg name="username" value="hiveuser" />
        <constructor-arg name="password" value="hivepassword" />
    </bean>
    <bean id="namedParameterJdbcHiveTemplate" name="namedParameterJdbcHiveTemplate" class="org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate">
        <constructor-arg ref="hiveDataSource"/>
    </bean>

    <hdp:hive-client-factory hive-data-source-ref="hiveDataSource" />

</beans>

原因分析 #

1. 版本兼容性问题：org.apache.hadoop.hive.service.HiveServerException是Hive 1.x及更早版本中的类，Spring Data Hadoop 2.5.0依赖的是旧版Hive客户端API。而你配置的是HiveServer2的JDBC驱动（org.apache.hive.jdbc.HiveDriver），如果你的Hive版本是2.x+，该类已被移除或重构，导致类加载失败。
2. Hive Tasklet实现机制：hdp:hive-tasklet底层使用Spring Data Hadoop的HiveTemplate，默认依赖旧版Hive客户端而非JDBC方式，和你手动测试的JDBC路径不一致。

解决方案 #

方案一：改用JDBC Tasklet替代Hive Tasklet（推荐） #

既然JDBC连接HiveServer2已经验证可行，直接实现自定义Tasklet使用NamedParameterJdbcTemplate执行SQL，避开Spring Data Hadoop的Hive模块兼容性问题。

自定义Tasklet代码

import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.stereotype.Component;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;

@Component
public class HivePartitionTasklet implements Tasklet {

    private final NamedParameterJdbcTemplate namedParameterJdbcHiveTemplate;

    public HivePartitionTasklet(NamedParameterJdbcTemplate namedParameterJdbcHiveTemplate) {
        this.namedParameterJdbcHiveTemplate = namedParameterJdbcHiveTemplate;
    }

    @Override
    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
        // 从JobExecutionContext获取日期参数
        String formattedOperationDate = chunkContext.getStepContext()
                .getJobExecutionContext()
                .getString("formattedOperationDate");
        
        // 拼接HDFS路径（可根据实际配置调整参数获取方式）
        String hdfsDataPath = chunkContext.getStepContext()
                .getJobParameters()
                .getString("hdfs.data.path");
        String activeItemsResource = chunkContext.getStepContext()
                .getJobParameters()
                .getString("hdfs.data.resource.active.items");
        String partitionLocation = String.format("%s/%s/%s", hdfsDataPath, activeItemsResource, formattedOperationDate);

        // 执行删除分区SQL
        String dropPartitionSql = "ALTER TABLE active_items DROP IF EXISTS PARTITION (batch_date=:batchDate)";
        namedParameterJdbcHiveTemplate.update(
                dropPartitionSql,
                Collections.singletonMap("batchDate", formattedOperationDate)
        );

        // 执行添加分区SQL
        String addPartitionSql = "ALTER TABLE active_items ADD PARTITION (batch_date=:batchDate) LOCATION :location";
        Map<String, Object> addParams = new HashMap<>();
        addParams.put("batchDate", formattedOperationDate);
        addParams.put("location", partitionLocation);
        namedParameterJdbcHiveTemplate.update(addPartitionSql, addParams);

        return RepeatStatus.FINISHED;
    }
}

配置Spring Batch Step

将原有的hdp:hive-tasklet替换为自定义Tasklet：

<batch:step id="addPartitionToActiveItemsTableScriptTaskletStep">
    <batch:tasklet ref="hivePartitionTasklet"/>
</batch:step>

方案二：调整依赖兼容旧版Hive API #

如果坚持使用hdp:hive-tasklet，需要确保依赖的Hive客户端版本与Spring Data Hadoop 2.5.0兼容：

1. 确认你的HiveServer2版本为1.x（如1.2.1），添加以下Maven依赖：

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-service</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.2.1</version>
    <exclusions>
        <exclusion>
            <groupId>org.eclipse.jetty.aggregate</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>

2. 若你的Hive版本是2.x+，此方案不适用，Spring Data Hadoop 2.5.0不兼容Hive 2.x的API结构，建议使用方案一。

内容的提问来源于stack exchange，提问作者prototype26