Spark 2.4.4如何拆分数组列内字符串并转为JSON格式？

原DataFrame结构 #

|-- students: array (nullable = true)
 |    |-- element: string (containsNull = true)

原始数据 #

+--------------------------------------------------+
|                                          students|
+--------------------------------------------------+
|                            [(Alice:20), (Bob:13)]|
|[(James:39), (Robert:29), (Kevin:31), (Andrew:48)]|
|                                    [(Richard:88)]|
+--------------------------------------------------+

目标输出 #

+-----------------------------------------------------------------------------------------------------------+
|                                                                                               json_student|
+-----------------------------------------------------------------------------------------------------------+
|[{"name":"Alice","age":20},{"name":"Bob","age":13}]                                                        |
|[{"name":"James","age":39},{"name":"Robert","age":29},{"name":"Kevin","age":31},{"name":"Andrew","age":48}]|
|[{"name":"Richard","age":88}]                                                                              |
+-----------------------------------------------------------------------------------------------------------+

实现代码 #

Spark 2.4+支持transform函数，可直接遍历数组处理每个元素，无需担心数组长度不一致问题，具体代码如下：

import org.apache.spark.sql.functions._

val resultDF = df.withColumn(
  "json_student",
  to_json(
    transform(
      col("students"),
      elem => struct(
        split(elem, ":")(0).alias("name"),
        split(elem, ":")(1).cast("int").alias("age")
      )
    )
  )
)

// 查看结果
resultDF.show(truncate = false)

代码说明 #

• transform(col("students"), ...)：遍历students数组的每一个元素，对每个字符串执行转换逻辑
• split(elem, ":")：将Alice:20这类字符串按冒号拆分为数组，取第0位作为name，第1位转为整数作为age，再用struct组合成结构化数据
• to_json(...)：将结构化的数组转换为JSON格式的字符串

内容的提问来源于stack exchange，提问作者fresh