Scala保存RDD至HDFS报错：空字符串引发NumberFormatException

Hey there, let’s tackle this NumberFormatException you’re hitting when trying to save your RDD to HDFS. The error message clearly points to an empty string being passed to Float.parseFloat—Java’s float parsing method can’t handle empty values, which is why your task is failing mid-execution.

Here’s how to diagnose and fix the issue: #

1. First, identify the problematic data

Before making fixes, it’s good to see exactly which records are causing the problem. Use a filter to isolate empty or whitespace-only strings in your RDD:

// Adjust the case statement to match your RDD's structure
val badRecords = yourRDD.filter { case (id, valueStr) =>
  valueStr.isEmpty || valueStr.trim.isEmpty
}
// Print a sample to inspect
badRecords.take(10).foreach(println)

This will show you the specific entries with empty values, helping you confirm if the issue is in your source data or a transformation step.

2. Choose a fix based on your data needs

You have a few options to handle empty strings, depending on whether you want to keep the records or discard them:

• Option 1: Filter out empty-value records
  If empty entries aren’t useful, just remove them before converting to floats:

val cleanedRDD = yourRDD.filter { case (id, valueStr) =>
  !valueStr.isEmpty && !valueStr.trim.isEmpty
}
// Convert to float and save
val floatRDD = cleanedRDD.map { case (id, valueStr) => (id, valueStr.toFloat) }
floatRDD.saveAsTextFile("hdfs://your-target-path")

• Option 2: Replace empty values with a default float
  If you need to retain the records, assign a default value (like 0.0f) to empty strings:

val safeFloatRDD = yourRDD.map { case (id, valueStr) =>
  val floatVal = if (valueStr.isEmpty || valueStr.trim.isEmpty) 0.0f else valueStr.toFloat
  (id, floatVal)
}
safeFloatRDD.saveAsTextFile("hdfs://your-target-path")

• Option 3: Use Option to explicitly mark missing values
  For better type safety, wrap your float values in an Option to distinguish between valid floats and missing data:

val optionalFloatRDD = yourRDD.map { case (id, valueStr) =>
  val floatOpt = if (valueStr.isEmpty || valueStr.trim.isEmpty) None else Some(valueStr.toFloat)
  (id, floatOpt)
}
// When saving, convert None to a readable string like "null"
optionalFloatRDD.map { case (id, opt) => s"$id,${opt.getOrElse("null")}" }
  .saveAsTextFile("hdfs://your-target-path")

3. Bonus: Add validation for valid float formats

Empty strings aren’t the only thing that can break float parsing. For extra robustness, validate that the string is actually a valid float using a regex:

val validFloatRegex = "^[-+]?[0-9]*\\.?[0-9]+([eE][-+]?[0-9]+)?$".r
val validatedRDD = yourRDD.filter { case (id, valueStr) =>
  val trimmed = valueStr.trim
  trimmed.nonEmpty && validFloatRegex.matches(trimmed)
}

内容的提问来源于stack exchange，提问作者Josué