升级Hadoop至3.1.2后hbase-testing-utility 2.2.3启动失败

目标 #

我需要从HDFS切换到s3a客户端，因此必须将Hadoop从2.8.5版本升级到至少3.1.2版本——因为要用到AssumedRoleCredentialProvider实现AWS访问。根据HBase文档的表5，Hadoop 3.1.2和HBase 2.2.3是兼容的。

问题 #

升级Hadoop客户端后，hbase-testing-util无法正常启动迷你集群。

测试代码：

utility = new HBaseTestingUtility(null);
utility.startMiniCluster();

执行mvn clean install时的异常 #

java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hdfs.server.namenode.FSDirectory
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:871)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:724)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1103)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
    at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
    at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:759)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:671)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:643)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1096)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1071)

执行mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2时的异常 #

java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
    at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:569)
    at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:550)
    at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:117)
    at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:425)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:160)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1313)
    at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1082)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:957)
    at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:889)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:805)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:671)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:643)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1096)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1071)

配置 #

原本基于Hadoop 2.8.5的配置可以正常运行，将Hadoop版本替换为3.1.2后失效：

<hbase.version>2.2.3</hbase.version>
<hadoop.version>2.8.5</hadoop.version>
...
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
  <version>${hadoop.version}</version>
  <exclusions>
    <exclusion>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.hbase</groupId>
  <artifactId>hbase-client</artifactId>
  <version>${hbase.version}</version>
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.hbase</groupId>
  <artifactId>hbase-testing-util</artifactId>
  <version>${hbase.version}</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <artifactId>servlet-api</artifactId>
      <groupId>javax.servlet</groupId>
    </exclusion>
    <exclusion>
      <groupId>org.glassfish</groupId>
      <artifactId>javax.el</artifactId>
    </exclusion>
  </exclusions>
</dependency>

已尝试操作 #

尝试过排除或手动引入不同版本的依赖，但都没解决问题。

解决方法 #

1. 修复NoClassDefFoundError异常

Hadoop 3.x的hadoop-client不再包含HDFS服务端相关类（比如namenode、datanode的核心类），但HBase测试工具启动迷你集群必须用到这些类。直接在pom里添加HDFS相关测试依赖：

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-hdfs</artifactId>
  <version>3.1.2</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-hdfs-client</artifactId>
  <version>3.1.2</version>
  <scope>test</scope>
</dependency>

2. 修复NoSuchMethodError异常

这个是Jetty版本冲突导致的——HBase 2.2.3和Hadoop 3.1.2依赖的Jetty版本不兼容，方法签名不匹配。解决步骤：

• 先排除hbase-testing-util里所有Jetty相关依赖：

<dependency>
  <groupId>org.apache.hbase</groupId>
  <artifactId>hbase-testing-util</artifactId>
  <version>${hbase.version}</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <artifactId>servlet-api</artifactId>
      <groupId>javax.servlet</groupId>
    </exclusion>
    <exclusion>
      <groupId>org.glassfish</groupId>
      <artifactId>javax.el</artifactId>
    </exclusion>
    <!-- 排除所有Jetty依赖，避免版本冲突 -->
    <exclusion>
      <groupId>org.eclipse.jetty</groupId>
      <artifactId>*</artifactId>
    </exclusion>
  </exclusions>
</dependency>

• 手动引入和Hadoop 3.1.2兼容的Jetty版本（9.3.24.v20180605）：

<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-server</artifactId>
  <version>9.3.24.v20180605</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-servlet</artifactId>
  <version>9.3.24.v20180605</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-servlets</artifactId>
  <version>9.3.24.v20180605</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-webapp</artifactId>
  <version>9.3.24.v20180605</version>
  <scope>test</scope>
</dependency>

3. 统一依赖版本管理

建议在pom的dependencyManagement里声明所有核心依赖的版本，彻底避免冲突：

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>3.1.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>3.1.2</version>
    </dependency>
    <!-- 导入Jetty的BOM，统一管理所有Jetty组件版本 -->
    <dependency>
      <groupId>org.eclipse.jetty</groupId>
      <artifactId>jetty-bom</artifactId>
      <version>9.3.24.v20180605</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

内容的提问来源于stack exchange，提问作者Milo