Spark中除构造函数传SparkConf外，是否有类似Hadoop Configured的配置访问方式？

Great question! Spark doesn’t ship with a direct, built-in equivalent to Hadoop’s Configured and Configurable interfaces, but we can achieve the same goal—letting classes easily access configuration values—through a few flexible approaches:

1. Use SparkContext/SparkSession to Access Configs #

The core Spark context objects (SparkContext or SparkSession) hold all your application’s configuration. You can pass a reference to these objects to your classes and pull config values directly from them:

class DataProcessor(spark: SparkSession) {
  // Fetch a custom config value from the SparkSession
  val inputPath = spark.conf.get("app.input.data.path")

  def process(): Unit = {
    println(s"Processing data from: $inputPath")
    // Your business logic here
  }
}

This is the most common pattern in Spark development, since SparkContext/SparkSession are already central to almost every Spark workflow.

2. Build Your Own Configured/Configurable-style Abstraction #

If you prefer Hadoop’s familiar pattern, you can easily roll your own set of traits/classes to mirror Configurable and Configured:

// Mirror Hadoop's Configurable interface
trait SparkConfigurable {
  def setConf(conf: SparkConf): Unit
  def getConf(): SparkConf
}

// Mirror Hadoop's Configured abstract class with default implementations
abstract class SparkConfigured extends SparkConfigurable {
  private var sparkConf: SparkConf = _

  override def setConf(conf: SparkConf): Unit = {
    this.sparkConf = conf
  }

  override def getConf(): SparkConf = {
    require(sparkConf != null, "SparkConf hasn't been initialized yet!")
    sparkConf
  }

  // Convenience method to fetch config values directly
  protected def getConfig(key: String): String = getConf().get(key)
}

// Example usage
class CustomETLJob extends SparkConfigured {
  def runETL(): Unit = {
    val outputPath = getConfig("app.output.data.path")
    // ETL logic here
  }
}

// Initialize in your main app
val conf = new SparkConf().setAppName("CustomETL").setMaster("local[*]")
val etlJob = new CustomETLJob()
etlJob.setConf(conf)
etlJob.runETL()

This approach feels right at home if you’re coming from a Hadoop background.

3. Use Dependency Injection (e.g., Guice) #

For larger, more complex applications, dependency injection can eliminate manual config/context passing entirely. You can bind SparkConf or SparkSession to your injector and inject it into any class that needs it:

import com.google.inject.{AbstractModule, Guice, Inject}

// Bind SparkConf in your module
val injector = Guice.createInjector(new AbstractModule() {
  override def configure(): Unit = {
    bind(classOf[SparkConf])
      .toInstance(new SparkConf().setAppName("DIExample"))
  }
})

// Inject SparkConf directly into your class
class ReportingService @Inject()(conf: SparkConf) {
  def generateReport(): Unit = {
    println(s"Generating report for app: ${conf.getAppName}")
  }
}

// Retrieve and use the service
val reportService = injector.getInstance(classOf[ReportingService])
reportService.generateReport()

This keeps your code clean and reduces boilerplate in large codebases.

One key thing to note: Unlike Hadoop’s mutable Configuration, Spark’s SparkConf is immutable once your application is running (you can only set values that don’t already exist with setIfMissing). Keep this in mind when designing your classes!

内容的提问来源于stack exchange，提问作者Majid Azimi