Java读取Parquet文件遇NoClassDefFoundError，求缺失依赖

问题详情 #

尝试用Java读取本地文件系统中的Parquet文件，代码实现如下：

public static void main(String[] args) throws IOException {
    String path = "a.parquet";
    var apr=new AvroParquetReader<GenericRecord>(new Path(path));
    try {
        apr.read();
    } catch (java.lang.Exception e) {
        throw new RuntimeException(e);
    }
}

使用的Maven依赖配置：

<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-common</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-column</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-encoding</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-hadoop</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-avro</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-format</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-format-structures</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>com.twitter</groupId>
        <artifactId>parquet-jackson</artifactId>
        <version>1.6.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.1</version>
    </dependency>
</dependencies>

运行程序时抛出如下异常：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/conf/ParquetConfiguration
    at org.apache.parquet.avro.AvroParquetReader.<init>(AvroParquetReader.java:114)
    at Main.main(Main.java:16)
Caused by: java.lang.ClassNotFoundException: org.apache.parquet.conf.ParquetConfiguration
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
    ... 2 more

解决方法 #

org.apache.parquet.conf.ParquetConfiguration类属于parquet-hadoop依赖包，但问题根源是你使用的hadoop-core 1.2.1版本过于陈旧，和Parquet 1.14.0版本兼容性差，导致依赖冲突或类加载失败。

具体修复步骤：

1. 移除旧的hadoop-core依赖
2. 添加兼容版本的hadoop-common依赖，建议使用和Parquet 1.14.0匹配的Hadoop 3.x版本，例如：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>3.3.6</version>
</dependency>

3. 确保parquet-hadoop依赖存在（你已添加，但需确认Maven已正确下载该依赖包，可清理本地Maven仓库后重新构建）

完成上述调整后，重新运行程序即可解决类缺失的问题。

内容的提问来源于stack exchange，提问作者Stepan Yakovenko