PySpark如何展平结构不同的嵌套数组并解决字段缺失报错

报错原因 #

你触发报错的核心原因是：部分name_开头的动态数组列，其元素结构体中不存在num字段，SQL表达式直接强制访问el.num时触发了字段不存在的异常。

解决方案 #

通过在Python侧提前校验每个数组列的结构体字段，动态构造转换逻辑，缺失字段自动补null，保证所有转换后的结构体结构统一，代码如下：

import pyspark.sql.functions as f

df = spark.read.json(
    sc.parallelize(
        [
            """{"id":1,"name_1_a":[{"date":2001,"val":1},{"date":2002,"val":2},{"date":2003,"val":3}],"name_10000_xvz":[{"date":2000,"val":30},{"date":2001,"val":31},{"date":2002,"val":32},{"date":2003,"val":33, "num":1}]}"""
        ]
    )
).select("id", "name_1_a", "name_10000_xvz")

names = [column for column in df.columns if column.startswith("name_")]
expressions = []

for name in names:
    # 读取当前数组列的结构体字段
    struct_fields = [field.name for field in df.schema[name].dataType.elementType.fields]
    # 动态构造STRUCT的字段部分，缺失num则补null
    struct_items = [f'"{name}" AS name', 'el.date', 'el.val']
    if 'num' in struct_fields:
        struct_items.append('el.num AS num')
    else:
        struct_items.append('NULL AS num')
    # 拼接生成TRANSFORM表达式
    transform_expr = f'TRANSFORM({name}, el -> STRUCT({", ".join(struct_items)}))'
    expressions.append(f.expr(transform_expr))

flatten_df = df.withColumn("flatten", f.flatten(f.array(*expressions))).selectExpr(
    "id", "inline(flatten)"
)

# 可选：将num列的null替换为空字符串匹配期望输出
flatten_df = flatten_df.fillna('', subset=['num'])
flatten_df.show()

执行结果 #

+---+--------------+----+---+---+
| id|          name|date|val|num|
+---+--------------+----+---+---+
|  1|      name_1_a|2001|  1|   |
|  1|      name_1_a|2002|  2|   |
|  1|      name_1_a|2003|  3|   |
|  1|name_10000_xvz|2000| 30|   |
|  1|name_10000_xvz|2001| 31|   |
|  1|name_10000_xvz|2002| 32|   |
|  1|name_10000_xvz|2003| 33|  1|
+---+--------------+----+---+---+

该方案无需硬编码动态列名，可自动适配最多10000个name_开头的数组列的处理需求。

内容的提问来源于stack exchange，提问作者Dan