Scala Spark代码循环执行时抛出NoSuchElementException异常求助

Hey there, let's figure out why that NoSuchElementException: None.get is popping up when calling FiloDriver.shutdown(), even though your aggregation loops run perfectly fine.

What's Causing the Issue? #

Looking at the error stack trace, the problem originates in FiloSetup.scala:31 when trying to access config.get. This means the FiloDriver's internal configuration wasn't properly initialized when you triggered the shutdown.

Breaking down your code flow:

• When iDate is not set to "0001-01-01 00:00:00", you skip the FiloDB read operation entirely. This means FiloDriver never gets initialized (its internal config remains None).
• When you call FiloDriver.shutdown() unconditionally in the finally block, the code tries to access the uninitialized config, which triggers the None.get exception.

Fixes to Try #

Here are two straightforward solutions to resolve this:

1. Only Shutdown FiloDriver If It Was Actually Used

Add a flag to track whether you initialized FiloDB, then conditionally call shutdown:

// Add this at the start of your try block
var filodbWasInitialized = false

val vBillMaxDate = if (iDate == "0001-01-01 00:00:00") {
  filodbWasInitialized = true // Mark that FiloDB was initialized
  hc.read.format("filodb.spark").option("database","*****").option("dataset","*****").load().filter(s"""yr_mon in ( ${vMonthCheck} )""").agg(max("*****").alias("max_date")).map {
    case Row (value) => value.asInstanceOf[java.sql.Timestamp];
  }.first()
} else {
  new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse(iDate)
}

// Update your finally block
finally{
  if (filodbWasInitialized) {
    FiloDriver.shutdown()
  }
  sc.stop()
  sys.exit(0)
}

2. Defensively Catch the Exception During Shutdown

If you prefer a safety net approach, wrap the shutdown call in a try-catch to handle cases where FiloDriver wasn't initialized:

finally{
  try {
    FiloDriver.shutdown()
  } catch {
    case e: java.util.NoSuchElementException =>
      println("Skipping FiloDriver shutdown: it was not initialized during this run.")
  }
  sc.stop()
  sys.exit(0)
}

Why This Works #

Both approaches avoid calling shutdown() on an uninitialized FiloDriver. The first solution is cleaner (only runs shutdown when necessary), while the second acts as a fallback for edge cases where initialization state might be unclear.

内容的提问来源于stack exchange，提问作者Rajdip