Spark Scala中List转DataFrame出现类型不匹配报错的解决方法

初始问题代码 #

import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.types.{StructField, StringType, DateType, BooleanType, StructType}
import scala.collection.mutable.ListBuffer

def createDf(list: ListBuffer[TestTO]): DataFrame = {
    val structArrLength = 6
    val cols: Array[StructField] = new Array[StructField](structArrLength)
    cols(0) = StructField("a", StringType, nullable=true)
    cols(2) = StructField("b", StringType, nullable=true)
    cols(3) = StructField("c", StringType, nullable=true)
    cols(4) = StructField("d", DateType, nullable=true)
    cols(5) = StructField("e", BooleanType, nullable=true)

    val schema = StructType(cols)
    val rddMap=sparkSession.sparkContext.parallelize(list)
    val rows = rddMap.map {  x => Row(Seq(x))}
    val df = sparkSession.createDataFrame(rows, schema)
    df.show(false)
}

报错信息 #

Caused by: java.lang.RuntimeException: scala.collection.immutable.$colon$colon is not a valid external type for schema of string

错误根因 #

• 构造Row时错误将整个对象封装为Seq传入，Row要求传入的参数和schema定义的字段数量、类型一一对应，不能直接传入集合类型
• 初始schema数组索引1未赋值，存在空字段定义问题

正确实现方案 #

手动构造Row+Schema版本 #

import org.apache.spark.sql.Dataset
import org.apache.spark.sql.Row
import org.apache.spark.sql.types.{StructField, StringType, StructType}
import scala.collection.mutable.ListBuffer

def createDf(list: ListBuffer[MaskScheduleTO]): Dataset[Row] = {
    val schema = createMaskSchema
    var data: ListBuffer[Row] = new ListBuffer[Row]
    list.foreach(x => data += Row(x.prop1, x.prop2))
    val rddMap = sparkSession.sparkContext.parallelize(data.toList)
    sparkSession.createDataFrame(rddMap, schema)
}

private def createMaskSchema = {
    StructType(List(
      StructField("prop1", StringType, nullable = true),
      StructField("prop2", StringType, nullable = true),
    ))
}

简化实现版本 #

如果自定义类是样例类（case class），可以直接通过隐式转换实现更简单的转换，无需手动构造Row和Schema：

import sparkSession.implicits._
// 直接将集合转为DataFrame，自动匹配样例类的字段名和类型
val df = list.toDF()

内容的提问来源于stack exchange，提问作者Mr. Richhariya