如何基于Delta表指定列创建Spark嵌套数组结构体Schema？

针对你给出的嵌套结构struct<a:string,b:array<struct<b1:string,b2:array<struct<b3:string,b4:int,b5:int>>,c:int>>>，可以通过从内到外逐层构建Schema的方式实现，以下是Python和Scala两种常用语言的实现代码：

Python 实现 #

from pyspark.sql.types import StructType, StructField, StringType, ArrayType, IntegerType

# 定义最内层b2数组的元素Schema
b2_element_schema = StructType([
    StructField("b3", StringType(), nullable=True),
    StructField("b4", IntegerType(), nullable=True),
    StructField("b5", IntegerType(), nullable=True)
])

# 定义b数组的元素Schema
b_element_schema = StructType([
    StructField("b1", StringType(), nullable=True),
    StructField("b2", ArrayType(b2_element_schema), nullable=True),
    StructField("c", IntegerType(), nullable=True)
])

# 最终reasonDetail列的完整Schema
reason_detail_schema = StructType([
    StructField("a", StringType(), nullable=True),
    StructField("b", ArrayType(b_element_schema), nullable=True)
])

# 若需定义包含reasonDetail列的完整Delta表Schema（示例）
delta_table_schema = StructType([
    StructField("id", StringType(), nullable=False),
    StructField("reasonDetail", reason_detail_schema, nullable=True)
])

Scala 实现 #

import org.apache.spark.sql.types._

// 最内层b2数组元素的Schema
val b2ElementSchema = StructType(Seq(
  StructField("b3", StringType, nullable = true),
  StructField("b4", IntegerType, nullable = true),
  StructField("b5", IntegerType, nullable = true)
))

// b数组元素的Schema
val bElementSchema = StructType(Seq(
  StructField("b1", StringType, nullable = true),
  StructField("b2", ArrayType(b2ElementSchema), nullable = true),
  StructField("c", IntegerType, nullable = true)
))

// reasonDetail列的完整Schema
val reasonDetailSchema = StructType(Seq(
  StructField("a", StringType, nullable = true),
  StructField("b", ArrayType(bElementSchema), nullable = true)
))

// 完整Delta表Schema示例
val deltaTableSchema = StructType(Seq(
  StructField("id", StringType, nullable = false),
  StructField("reasonDetail", reasonDetailSchema, nullable = true)
))

注意事项 #

• 嵌套Schema必须从最内层的结构开始定义，再逐层向外组合，避免层级混乱
• nullable参数可根据业务需求设置为true（允许空值）或false（非空约束）
• 创建Delta表时，可通过spark.createDataFrame(data, schema=delta_table_schema)或DeltaTable.create().schema(deltaTableSchema)...等方式使用该Schema

内容的提问来源于stack exchange，提问作者Saswat Ray