Scala测试Mock Hadoop FileSystem遇冻结类异常，求解决方案

我在Scala测试中尝试Mock Hadoop的FileSystem类，编写了如下测试代码：

import java.net.URI
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.junit.Test
import org.junit.runner.RunWith
import org.mockito.ArgumentMatchers.any
import org.mockito.Mockito._
import org.powermock.api.mockito.PowerMockito
import org.powermock.core.classloader.annotations.PrepareForTest
import org.powermock.modules.junit4.PowerMockRunner

@RunWith(classOf[PowerMockRunner])
@PrepareForTest(Array(classOf[FileSystem]))  
class SimpleFileSystemTest {

  @Test
  def testMockStaticFileSystem(): Unit = {
    // Mock static method FileSystem.get()
    PowerMockito.mockStatic(classOf[FileSystem])
    val mockFileSystem = mock(classOf[FileSystem])

    // Define behavior of FileSystem.get()
    when(FileSystem.get(any(classOf[URI]), any(classOf[Configuration]))).thenReturn(mockFileSystem)

    // Add simple assertions to verify the behavior
    val uri = new URI("s3a://bucket")
    val conf = new Configuration()

    // Verify that the mocked FileSystem is returned
    val fileSystem = FileSystem.get(uri, conf)
    assert(fileSystem eq mockFileSystem)

    // Verify that the static method was called
    PowerMockito.verifyStatic(classOf[FileSystem], times(1))
    FileSystem.get(uri, conf)
  }
}

运行时抛出以下异常：

org.mockito.exceptions.base.MockitoException: 
Mockito cannot mock this class: class org.apache.hadoop.fs.FileSystem.

Mockito can only mock non-private & non-final classes.
If you're not sure why you're getting this error, please report to the mailing list.


Java               : 1.8
JVM vendor name    : Amazon.com Inc.
JVM vendor version : 25.422-b05
JVM name           : OpenJDK 64-Bit Server VM
JVM version        : 1.8.0_422-b05
JVM info           : mixed mode
OS name            : Mac OS X
OS version         : 14.7


Underlying exception : java.lang.IllegalStateException: Failed to transform class with name org.apache.hadoop.fs.FileSystem$Cache. Reason: org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen

    at SimpleFileSystemTest.testMockStaticFileSystem(SimpleFileSystemTest.scala:19)
    at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
    at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
    at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
    at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
    at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: Failed to transform class with name org.apache.hadoop.fs.FileSystem$Cache. Reason: org.apache.hadoop.fs.FileSystem$Cache$Key class is frozen
    at org.powermock.core.classloader.javassist.JavassistMockClassLoader.defineAndTransformClass(JavassistMockClassLoader.java:119)
    at org.powermock.core.classloader.MockClassLoader.loadMockClass(MockClassLoader.java:174)
    at org.powermock.core.classloader.MockClassLoader.loadClassByThisClassLoader(MockClassLoader.java:102)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass1(DeferSupportingClassLoader.java:147)
    at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass(DeferSupportingClassLoader.java:98)

依赖配置如下：

libraryDependencies ++= Seq(
      "org.slf4j" % "slf4j-api" % "2.0.7" % "provided",
      "com.google.guava" % "guava" % "32.1.2-jre" % "test",
      "org.scalatest" %% "scalatest" % "3.2.16" % "test",
      "org.scalatestplus" %% "mockito-3-4" % "3.2.10.0" % "test",
      "org.powermock" % "powermock-module-junit4" % "2.0.9" % "test",
      "org.powermock" % "powermock-api-mockito2" % "2.0.9" % "test" exclude("org.mockito", "mockito-core"),
      "org.mockito" % "mockito-core" % "3.12.4" % "test",
)

1. 升级PowerMock版本 #

PowerMock 2.0.9对新版本Hadoop的类处理存在兼容性问题，建议升级到PowerMock 2.0.10+，新版本修复了不少类转换的bug，适配Java 8环境更稳定。

修改依赖配置：

libraryDependencies ++= Seq(
      // 其他依赖保持不变
      "org.powermock" % "powermock-module-junit4" % "2.0.10" % "test",
      "org.powermock" % "powermock-api-mockito2" % "2.0.10" % "test" exclude("org.mockito", "mockito-core"),
      "org.mockito" % "mockito-core" % "3.12.4" % "test",
)

2. 调整@PrepareForTest的范围 #

异常提示FileSystem$Cache$Key类被冻结，说明PowerMock未正确准备相关内部类。可以扩展@PrepareForTest的范围，显式包含内部类：

@PrepareForTest(Array(classOf[FileSystem], classOf[FileSystem.Cache], classOf[FileSystem.Cache.Key]))

或者使用通配符简化配置：

@PrepareForTest(Array("org.apache.hadoop.fs.FileSystem*"))

3. 替换Mock方案：使用Hadoop官方MockFileSystem #

如果PowerMock的问题始终无法解决，直接使用Hadoop官方提供的MockFileSystem类，它是专为测试设计的实现类，无需静态Mock：

步骤： #

1. 添加Hadoop测试依赖：

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "你的Hadoop版本" % "test" classifier "tests"

2. 修改测试代码：

import java.net.URI
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, MockFileSystem}
import org.junit.Test

class SimpleFileSystemTest {

  @Test
  def testFileSystemBehavior(): Unit = {
    val conf = new Configuration()
    // 配置使用MockFileSystem
    conf.set("fs.s3a.impl", classOf[MockFileSystem].getName)
    
    val fileSystem = FileSystem.get(new URI("s3a://bucket"), conf)
    // 验证返回的是MockFileSystem实例
    assert(fileSystem.isInstanceOf[MockFileSystem])
    
    // 直接调用MockFileSystem方法模拟业务场景
    val mockFs = fileSystem.asInstanceOf[MockFileSystem]
    mockFs.create(new URI("s3a://bucket/test.txt"), true)
    assert(mockFs.exists(new URI("s3a://bucket/test.txt")))
  }
}

4. 检查Hadoop版本兼容性 #

确保Hadoop版本与PowerMock、Mockito版本匹配。若使用Hadoop 3.x+版本，需搭配PowerMock 2.0.10+，旧版本PowerMock无法正确处理新版本Hadoop的内部类结构。

内容的提问来源于stack exchange，提问作者rahman