Flink+Hudi写入S3报HFileBootstrapIndex初始化失败求助

错误日志 #

java.lang.NoClassDefFoundError: Could not initialize class org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Unknown Source)
    at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55)
    at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79)
    at org.apache.hudi.common.bootstrap.index.BootstrapIndex.getBootstrapIndex(BootstrapIndex.java:163)
    at org.apache.hudi.common.table.view.AbstractTableFileSystemView.init(AbstractTableFileSystemView.java:118)
    at org.apache.hudi.common.table.view.HoodieTableFileSystemView.init(HoodieTableFileSystemView.java:113)
    at org.apache.hudi.common.table.view.HoodieTableFileSystemView.<init>(HoodieTableFileSystemView.java:107)
    at org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:177)
    at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$createViewManager$5fcdabfe$1(FileSystemViewManager.java:272)
    at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$1(FileSystemViewManager.java:115)
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
    at org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:114)
    at org.apache.hudi.table.HoodieTable.getSliceView(HoodieTable.java:320)
    at org.apache.hudi.sink.partitioner.profile.DeltaWriteProfile.getFileSystemView(DeltaWriteProfile.java:93)
    at org.apache.hudi.sink.partitioner.profile.WriteProfile.<init>(WriteProfile.java:120)
    at org.apache.hudi.sink.partitioner.profile.DeltaWriteProfile.<init>(DeltaWriteProfile.java:44)
    at org.apache.hudi.sink.partitioner.profile.WriteProfiles.getWriteProfile(WriteProfiles.java:75)
    at org.apache.hudi.sink.partitioner.profile.WriteProfiles.lambda$singleton$0(WriteProfiles.java:64)
    at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
    at org.apache.hudi.sink.partitioner.profile.WriteProfiles.singleton(WriteProfiles.java:63)
    at org.apache.hudi.sink.partitioner.BucketAssigners.create(BucketAssigners.java:56)
    at org.apache.hudi.sink.partitioner.BucketAssignFunction.open(BucketAssignFunction.java:122)
    at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:101)
    at org.apache.flink.streaming.api.operators.KeyedProcessOperator.open(KeyedProcessOperator.java:55)
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
    at java.base/java.lang.Thread.run(Unknown Source)
    Suppressed: java.lang.NullPointerException
        at org.apache.hudi.sink.partitioner.BucketAssignFunction.close(BucketAssignFunction.java:247)
        at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:41)
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.close(AbstractUdfStreamOperator.java:115)
        at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163)
        at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1043)
        at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255)
        at org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72)
        at org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:951)
        at org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$0(Task.java:934)
        at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
        at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:934)

当前Hudi配置 #

hoodie.compaction.payload.class=org.apache.hudi.common.model.EventTimeAvroPayload
hoodie.table.type=MERGE_ON_READ
hoodie.table.partition.fields=eventId,eventName
hoodie.table.cdc.enabled=false
hoodie.archivelog.folder=archived
hoodie.timeline.layout.version=1
hoodie.table.checksum=3942898242
hoodie.datasource.write.drop.partition.columns=false
hoodie.table.recordkey.fields=event_id,event_timestamp
hoodie.table.name=sink_table
hoodie.compaction.record.merger.strategy=eeb8d96f-b1e4-49fd-bbf8-28ac514178e5
hoodie.datasource.write.hive_style_partitioning=false
hoodie.table.keygenerator.class=org.apache.hudi.keygen.ComplexAvroKeyGenerator
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.table.version=5

已尝试操作 #

• 添加hudi-flink1.16-bundle-0.13.1到类路径，出现多个NoClassDefFoundError
• 改为手动添加单个依赖，问题未解决

解决方案建议 #

1. 对齐Flink与Hudi的版本兼容性 #

Hudi 0.13.1官方仅提供适配Flink 1.16的bundle包，直接用于Flink 1.17会存在API不兼容问题。两种处理方式：

• 降级Flink到1.16.x版本，与Hudi 0.13.1的bundle版本匹配
• 升级Hudi到0.14.0及以上版本，该版本开始提供适配Flink 1.17的bundle包

2. 补充HBase相关依赖 #

HFileBootstrapIndex的初始化失败通常是缺少HBase依赖导致的。若必须使用当前版本组合，需添加以下HBase依赖（以Maven为例）：

<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-client</artifactId>
    <version>2.4.11</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-common</artifactId>
    <version>2.4.11</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-server</artifactId>
    <version>2.4.11</version>
    <scope>provided</scope>
</dependency>

提交Flink作业时，需将这些依赖打包进作业JAR，或通过--jar参数传入集群。

3. 禁用Bootstrap索引（若未使用该功能） #

如果未用到Hudi的Bootstrap数据导入功能，可显式配置禁用相关索引，避免加载HFileBootstrapIndex：
在Hudi配置中添加：

hoodie.bootstrap.index.class=org.apache.hudi.common.bootstrap.index.NoOpBootstrapIndex

4. 使用官方完整Bundle包 #

避免手动拆分依赖，优先使用官方提供的对应Flink版本的Hudi bundle包，确保所有依赖的兼容性和完整性，减少依赖缺失或冲突的概率。

内容的提问来源于stack exchange，提问作者user3497321