Spark Schema嵌入外部库时Scala反射实例化类的问题

Let's tackle your reflection issue step by step—your goal of instantiating a HydraTargetTable subclass from a class name string is totally achievable, but there are a few pitfalls in your current code that we need to address.

First, What's Wrong with Your Current Code? #

Your existing reflection code uses Class.newInstance(), which is deprecated in Java (and by extension, Scala) because it swallows exceptions and doesn't handle constructor failures gracefully. Additionally, Scala's class compilation behaves differently than Java when it comes to constructors, so relying on this old method can lead to unexpected errors.

Solution 1: Use Java Reflection (Simpler, No Extra Dependencies) #

We'll explicitly fetch the no-arg constructor and use it to instantiate the class, which is the recommended replacement for newInstance().

Here's the revised hydraTableBuilder method:

object ReflectionTestApp {

  def hydraTableBuilder(name: String): HydraTargetTable = {
    // Fully qualified class name (ensure your package path is correct)
    val fullClassName = "hearsay.hydra.dataflow.api." + name
    // Load the class
    val clazz = Class.forName(fullClassName)
    // Get the public no-arg constructor
    val noArgConstructor = clazz.getConstructor()
    // Instantiate and cast to your trait type
    noArgConstructor.newInstance().asInstanceOf[HydraTargetTable]
  }

  def main(args: Array[String]): Unit = {
    hydraTableBuilder("Table1").inputTables.foreach(println)
  }
}

Key Notes for This Approach:

• Ensure your class has a public no-arg constructor: Your Table1 class already fits this since it doesn't define any constructor parameters (Scala generates a default no-arg constructor for top-level classes with no parameters).
• Double-check the fully qualified class name: Typos in the package or class name will cause a ClassNotFoundException—verify that hearsay.hydra.dataflow.api.Table1 is exactly the path to your compiled class.
• Classpath visibility: In Spark, make sure the JAR containing your Table1 class is included in your application's classpath (either via --jars when submitting or as a dependency in your build file).

Solution 2: Use Scala's Native Reflection API (More Scala-idiomatic) #

If you prefer an approach tailored to Scala's type system, you can use Scala's built-in reflection library. This is especially useful if you're working with more complex Scala constructs (like case classes or nested classes).

First, add the Scala reflect dependency to your build (for sbt):

libraryDependencies += "org.scala-lang" % "scala-reflect" % scalaVersion.value

Then, rewrite your builder using Scala reflection:

import scala.reflect.runtime.universe._

object ReflectionTestApp {

  def hydraTableBuilderScala(name: String): HydraTargetTable = {
    val fullClassName = "hearsay.hydra.dataflow.api." + name
    // Create a mirror using the current class loader
    val mirror = runtimeMirror(getClass.getClassLoader)
    // Load the class symbol
    val classSymbol = mirror.staticClass(fullClassName)
    // Get a mirror for the class
    val classMirror = mirror.reflectClass(classSymbol)
    // Fetch the primary constructor (no-arg in your case)
    val primaryConstructor = classSymbol.primaryConstructor.asMethod
    // Instantiate the class and cast
    classMirror.reflectConstructor(primaryConstructor)().asInstanceOf[HydraTargetTable]
  }

  def main(args: Array[String]): Unit = {
    hydraTableBuilderScala("Table1").inputTables.foreach(println)
  }
}

Why This Works Better for Scala:

• It handles Scala-specific class structures more naturally (like case class apply methods, nested classes, etc.).
• It provides better type safety when working with Scala's type system, compared to Java reflection.

Final Checks to Avoid Common Errors #

1. Verify trait implementation: Ensure Table1 correctly implements all abstract members of HydraTargetTable (your current code does this, but double-check if you modify the trait later).
2. Avoid package-private classes: If you change the visibility of Table1 to package-private, reflection will fail unless you adjust access permissions (stick to public top-level classes for simplicity here).
3. Spark classloader considerations: In distributed Spark environments, make sure your class is available on both the driver and executors. Using --packages or bundling your code into an uber-JAR can help with this.

内容的提问来源于stack exchange，提问作者Shivakanth Komatreddy