Hive生产环境调用Jar报Protobuf类型不兼容错误求助

问题描述 #

团队在Hive生产环境部署流程：通过HQL调用Jar并传入输入数据，Jar返回结果。Java代码在终端运行完全正常，但在Hive中偶尔抛出VerifyError，错误栈如下：

Vertex failed, vertexName=Map 1, vertexId=vertex_1672281536650_0001_1_00, diagnostics=[Task failed, taskId=task_1672281536650_0001_1_00_000003, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1672281536650_0001_1_00_000003_0:java.lang.RuntimeException: java.lang.RuntimeException: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;)Ljava/lang/Object; @2: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @2
    flags: { }
    locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
    stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0x0000000: 2a2b b600 23b0                          

    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
    at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
    at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
    at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
    at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;)Ljava/lang/Object; @2: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @2
    flags: { }
    locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
    stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0x0000000: 2a2b b600 23b0

堆栈跟踪起始为 at edu.stanford.nlp.simple.Document.<init>(Document.java:400)，Jar依赖Stanford Core NLP，Protobuf为其间接依赖。已尝试升级到Stanford Core NLP v4.5.1，分别搭配Protobuf v3.21.12和Core NLP pom指定的v3.19.2，问题仍存在。

解决方案 #

1. 排查Hive/Tez环境的Protobuf版本冲突 #

Hive、Tez或Hadoop组件可能自带旧版本Protobuf Jar，与Jar中的版本冲突导致类加载异常：

• 在Hive节点执行hive --auxpath | grep protobuf，或检查$HIVE_HOME/lib、$TEZ_HOME/lib目录下的protobuf相关Jar，确认环境版本。
• 采用Jar隔离方案：
  • 用maven-shade-plugin打包Jar时，将Protobuf类重命名到自定义命名空间，避免与环境类冲突。
  • 在HQL中通过ADD JAR指定Jar路径，或配置hive.aux.jars.path确保自定义Jar优先加载。

2. 调整Tez任务的类加载顺序 #

Hive+Tez默认类加载顺序可能让环境Jar优先于自定义依赖加载：

• 修改Tez配置tez.cluster.additional.classpath，将自定义Jar路径放在最前端。
• 执行HQL前添加配置：SET hive.tez.container.classpath=/path/to/your/jar/*;，强制指定容器类路径。

3. 确保Jar打包无依赖冲突 #

• 用Maven依赖树分析工具排查多版本Protobuf：

  mvn dependency:tree | grep protobuf
  

• 显式排除所有间接Protobuf依赖，指定统一版本，并将Protobuf类打入Jar（不依赖环境）。
• 配置maven-shade-plugin重命名Protobuf包：

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.4.1</version>
    <executions>
      <execution>
        <phase>package</phase>
        <goals>
          <goal>shade</goal>
        </goals>
        <configuration>
          <relocations>
            <relocation>
              <pattern>com.google.protobuf</pattern>
              <shadedPattern>your.custom.prefix.protobuf</shadedPattern>
            </relocation>
          </relocations>
        </configuration>
      </execution>
    </executions>
  </plugin>
  

4. 优化任务资源配置 #

偶尔报错可能与Tez容器资源不足导致的类加载异常有关：

• 增加容器内存：SET tez.container.size=4096;（根据集群情况调整）。
• 调整Hive并行度：SET hive.exec.parallel=true;，避免任务堆积引发资源竞争。

5. 严格匹配Core NLP与Protobuf的兼容版本 #

Stanford Core NLP对Protobuf版本有明确依赖要求，不要随意升级：

• 遵循官方文档，v4.5.1推荐使用Protobuf v3.19.2，打包时强制排除其他来源的Protobuf：

  <dependency>
    <groupId>edu.stanford.nlp</groupId>
    <artifactId>stanford-corenlp</artifactId>
    <version>4.5.1</version>
    <exclusions>
      <exclusion>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
      </exclusion>
    </exclusions>
  </dependency>
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.19.2</version>
  </dependency>
  

内容的提问来源于stack exchange，提问作者Santanu Bhattacharjee