如何将数组列'languages1'与'languages2'合并为结构体数组列？

需求说明 #

现有两个数组列languages1和languages2，示例数据如下：

languages1 = ["Java1","Scala1","C++1"]
languages2 = ["Java2","Scala2","C++2"]

需要将二者合并为名为lang的结构体数组列，最终格式要求如下：

lang:
[ data:{
   language:Java1,
   languages2: Java2
   },
  data:{
   language:Scala1,
   languages2: Scala2
   },
  data:{
   language:C++1,
   languages2: C++2
   }
]

实现方案 #

PySpark代码实现 #

from pyspark.sql import SparkSession
from pyspark.sql.functions import zip_with, struct

# 初始化Spark会话
spark = SparkSession.builder.appName("MergeArrayToStruct").getOrCreate()

# 构造测试数据
test_data = [
    (["Java1","Scala1","C++1"], ["Java2","Scala2","C++2"])
]
df = spark.createDataFrame(test_data, schema=["languages1", "languages2"])

# 合并数组生成目标结构体数组列
df = df.withColumn(
    "lang",
    zip_with(
        df.languages1,
        df.languages2,
        lambda l1, l2: struct(l1.alias("language"), l2.alias("languages2")).alias("data")
    )
)

# 输出结果
df.select("lang").show(truncate=False)

Spark SQL语句实现 #

SELECT
  zip_with(
    languages1,
    languages2,
    (l1, l2) -> named_struct('data', named_struct('language', l1, 'languages2', l2))
  ) AS lang
FROM your_table_name

逻辑说明 #

• zip_with函数会按索引位置一一对应两个数组的元素，对每一组元素执行自定义逻辑
• 通过struct（Spark SQL中用named_struct）将对应位置的元素封装为嵌套结构体，外层结构体别名设为data，内层分别对应language和languages2字段

内容的提问来源于stack exchange，提问作者HAEK