使用PySpark移除嵌套JSON对象数组（empval）中的id字段

需要使用PySpark移除嵌套JSON键值对对象数组（empval字段）中的id字段。

输入数据集 #

+----------+--------+----------------------------------------------------------------------------------------------------------+
| empno    | empcode| empval                                                                                                   |
+----------+--------+----------------------------------------------------------------------------------------------------------+
| employee1| 100DRE | [{"id": "123", "key1": "value1", "key2": "value2"}, {"id": "234", "key1": "te", "key2": "value2"}, {"id": "345", "key1": "grtregert", "key2": "value2"}] |
+----------+--------+----------------------------------------------------------------------------------------------------------+

预期输出 #

+----------+--------+---------------------------------------------------------------------------------------------------------------------+
| empno    | empcode| newColumn                                                                                                           |
+----------+--------+---------------------------------------------------------------------------------------------------------------------+
| employee1| 100DRE | [{"key1": "value1", "key2": "value2"}, {"key1": "te", "key2": "value2"}, {"key1": "grtregert", "key2": "value2"}]|
+----------+--------+---------------------------------------------------------------------------------------------------------------------+

解决方案 #

利用PySpark的transform函数遍历数组内的每个JSON对象，通过drop方法移除id字段，生成目标列。

完整代码示例 #

from pyspark.sql import SparkSession
from pyspark.sql.functions import transform, col

# 初始化Spark会话
spark = SparkSession.builder.appName("RemoveIdFromNestedArray").getOrCreate()

# 构造输入数据集
input_data = [
    ("employee1", "100DRE", [
        {"id": "123", "key1": "value1", "key2": "value2"},
        {"id": "234", "key1": "te", "key2": "value2"},
        {"id": "345", "key1": "grtregert", "key2": "value2"}
    ])
]
df = spark.createDataFrame(input_data, schema=["empno", "empcode", "empval"])

# 处理数组字段，移除每个对象的id键
result_df = df.withColumn(
    "newColumn",
    transform(col("empval"), lambda obj: obj.drop("id"))
).drop("empval")

# 打印结果
result_df.show(truncate=False)

关键逻辑说明 #

• transform(col("empval"), lambda obj: obj.drop("id"))：对empval数组中的每个JSON对象执行drop("id")操作，剔除指定字段。
• drop("empval")：可选操作，用于移除原数组字段，仅保留处理后的newColumn。

内容的提问来源于stack exchange，提问作者user23493242