Spark中如何将嵌套JSON属性展开为DataFrame的独立列

问题说明 #

你的代码已正确解析JSON并生成包含嵌套Struct的DataFrame，但show()方法默认会将Struct类型字段作为整体显示，不会展开其内部属性。要让所有属性都作为独立列显示，只需在select时显式展开Struct的子字段。

修正后的代码 #

方法1：直接展开Struct子字段（最简洁）

先修正原JSON字符串的语法错误（末尾多了一个}]），再通过.*语法展开Struct的所有子字段：

val payload = """{"invoice":[{"currency_code":"USD", "customer_detail":{"partner_name":"name","bill_to_city":"Arlington","bill_to_state":"Texas","bill_to_country":"US","contact_phone":"123.456.7890","country_name":"United States"}}]}"""

val df = spark.read.option("multiline", "true")
    .json(Seq(payload).toDS())
    .selectExpr("explode(invoice) as invoice")
    .select(
      "invoice.currency_code",  // 提取顶级字段
      "invoice.customer_detail.*"  // 展开Struct的所有子字段
    )

df.show(false)

方法2：手动指定所有字段（适合自定义列名场景）

如果需要给子字段重命名，可以逐个显式指定：

val df = spark.read.option("multiline", "true")
    .json(Seq(payload).toDS())
    .selectExpr("explode(invoice) as invoice")
    .select(
      "invoice.currency_code",
      "invoice.customer_detail.partner_name",
      "invoice.customer_detail.bill_to_city",
      "invoice.customer_detail.bill_to_state",
      "invoice.customer_detail.bill_to_country",
      "invoice.customer_detail.contact_phone",
      "invoice.customer_detail.country_name"
    )

df.show(false)

效果说明 #

执行修正后的代码后，show()会输出所有独立列：

+-------------+------------+-----------+------------+---------------+--------------+---------------+
|currency_code|partner_name|bill_to_city|bill_to_state|bill_to_country|contact_phone |country_name   |
+-------------+------------+-----------+------------+---------------+--------------+---------------+
|USD          |name        |Arlington  |Texas       |US             |123.456.7890   |United States  |
+-------------+------------+-----------+------------+---------------+--------------+---------------+

内容的提问来源于stack exchange，提问作者EXEC1