Hive ORC表分片生成失败问题排查与解决求助

环境信息 #

• Hadoop 3.3.4
• Hive 3.1.3
• Tez 0.9.2

问题现象 #

通过CTAS创建带分片的ORC表后执行查询，抛出ORC split generation failed异常。部分场景下执行表合并操作可临时解决问题，但仍有部分场景问题复现。

操作步骤 #

CREATE TABLE ClaimsOrc STORED AS ORC
AS
SELECT *
FROM ClaimsImport;

SELECT COUNT(*) FROM ClaimsOrc WHERE ClaimID LIKE '%8%';

报错详情 #

Vertex failed, vertexName=Map 1, vertexId=vertex_1667735849290_0008_6_00, diagnostics=[Vertex vertex_1667735849290_0008_6_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: claimsorc initializer failed, vertex=vertex_1667735849290_0008_6_00 [Map 1], java.lang.RuntimeException: ORC split generation failed with exception: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1851)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1939)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:519)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:765)
        at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1790)

临时解决方法 #

执行表合并操作后，查询可正常执行：

ALTER TABLE ClaimsOrc CONCATENATE;
OK
Time taken: 11.673 seconds

SELECT COUNT(*) FROM ClaimsOrc WHERE ClaimID LIKE '%8%';
OK
1463419
Time taken: 7.446 seconds, Fetched: 1 row(s)

初步推测与需求 #

初步判断初始CTAS查询在计算分片时存在问题，但CONCATENATE仅能解决部分场景的问题，现寻求彻底修复该问题的方案。

补充信息 #

• 执行DESCRIBE EXTENDED ClaimsOrc;可确认ClaimsOrc为ORC表。
• 源表ClaimsImport包含约24个gzip压缩的竖线分隔文件。
• CONCATENATE操作前，ClaimsOrc表包含约24个文件。
• CONCATENATE操作后，ClaimsOrc表仅包含3个文件分片。
• CONCATENATE操作前，使用orcfiledump命令检测ORC文件未发现错误。

内容的提问来源于stack exchange，提问作者Patrick Tucci