Spark Scala嵌套Struct列表遍历与转换技术求助

报错的核心原因是a.b.c是Struct类型而非Map类型，你用x.key是把它当成了键值对结构，但Struct的字段是固定命名的（比如unitedStates、unitedKingdom），不存在名为key的字段，因此触发分析异常。

因为国家字段是动态的，需要先将Struct转为键值对Map，再逐层展开关联员工与国家，最后按角色分组聚合。

步骤1：将Struct转为键值对Map #

利用map_keys获取Struct的所有字段名（即国家名），再通过transform将每个字段名与对应值封装为Struct，最后用map_from_entries转为Map结构：

import org.apache.spark.sql.functions._

val dfWithMap = df.withColumn("country_map", 
  map_from_entries(
    transform(
      map_keys($"a.b.c"), 
      key => struct(key as "country_name", $"a.b.c".getField(key) as "emp_data")
    )
  )
)

步骤2：展开Map拆分国家与员工数据 #

通过explode展开Map，将国家名和对应的员工结构拆分为单独列：

val dfCountryEmp = dfWithMap.selectExpr("explode(country_map) as (country, emp_struct)")
  .select($"country", $"emp_struct.employees")

步骤3：展开员工数组并按角色分组 #

展开员工数组，按角色分组后聚合员工姓名与所属国家：

val finalResult = dfCountryEmp.select($"country", explode($"employees") as "employee")
  .groupBy($"employee.role")
  .agg(collect_list(struct($"employee.name", $"country")) as "employee_country")

假设原始DataFrame的Schema如下：

root
 |-- a: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- b: struct (nullable = true)
 |    |    |    |-- c: struct (nullable = true)
 |    |    |    |    |-- unitedStates: struct (nullable = true)
 |    |    |    |    |    |-- employees: array (nullable = true)
 |    |    |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |    |    |-- name: string (nullable = true)
 |    |    |    |    |    |    |    |-- role: string (nullable = true)
 |    |    |    |    |-- unitedKingdom: struct (nullable = true)
 |    |    |    |    |    |-- employees: array (nullable = true)
 |    |    |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |    |    |-- name: string (nullable = true)
 |    |    |    |    |    |    |    |-- role: string (nullable = true)

完整可运行代码：

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object SparkNestedStructProcessing {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("NestedStructProcessing")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

    // 模拟原始数据
    val data = Seq(
      (Seq(
        (
          (
            (
              (Seq(("Alice", "Engineer"), ("Bob", "Manager"))),
              (Seq(("Charlie", "Engineer"), ("David", "Designer")))
            )
          )
        )
      ))
    )

    val df = data.toDF("a")
      .withColumn("a", $"a".cast("array<struct<b:struct<c:struct<unitedStates:struct<employees:array<struct<name:string,role:string>>>,unitedKingdom:struct<employees:array<struct<name:string,role:string>>>>>>>"))

    // 步骤1：转Struct为Map
    val dfWithMap = df.withColumn("country_map", 
      map_from_entries(
        transform(
          map_keys($"a.b.c"), 
          key => struct(key as "country_name", $"a.b.c".getField(key) as "emp_data")
        )
      )
    )

    // 步骤2：展开Map
    val dfCountryEmp = dfWithMap.selectExpr("explode(country_map) as (country, emp_struct)")
      .select($"country", $"emp_struct.employees")

    // 步骤3：分组聚合
    val finalResult = dfCountryEmp.select($"country", explode($"employees") as "employee")
      .groupBy($"employee.role")
      .agg(collect_list(struct($"employee.name", $"country")) as "employee_country")

    finalResult.show(false)
  }
}

• map_keys：提取Struct的所有字段名，解决动态国家字段的获取问题
• map_from_entries：将键值对Struct集合转为Map，实现Struct到动态键值结构的转换
• explode：分别展开Map和数组，实现一对多的关联拆分
• collect_list：按角色分组后，聚合每个员工的姓名与所属国家信息

内容的提问来源于stack exchange，提问作者jypyg