PySpark DataFrame使用to_json()函数转换整行为JSON列的实现

我来帮你把这段PySpark的内容整理成清晰的Markdown格式，顺便解释下实现逻辑~

首先来看我们的原始PySpark DataFrame：

>>> df.show(10,False)
+-----+----+---+------+
|id   |name|age|salary|
+-----+----+---+------+
|10001|alex|30 |75000 |
|10002|bob |31 |80000 |
|10003|deb |31 |80000 |
|10004|john|33 |85000 |
|10005|sam |30 |75000 |
+-----+----+---+------+

我们的需求是把这个DataFrame的每一行转换成一个新的JSON格式列jsonCol，对应的实现代码如下：

>>> newDf1 = df.withColumn("jsonCol", to_json(struct([df[x] for x in df.columns])))
>>> newDf1.show(10,False)
+-----+----+---+------+----------------------------------------------------+
|id   |name|age|salary|jsonCol                                             |
+-----+----+---+------+----------------------------------------------------+
|10001|alex|30 |75000 |{"id":"10001","name":"alex","age":"30","salary":"75000"}|
|10002|bob |31 |80000 |{"id":"10002","name":"bob","age":"31","salary":"80000"}|
|10003|deb |31 |80000 |{"id":"10003","name":"deb","age":"31","salary":"80000"}|
|10004|john|33 |85000 |{"id":"10004","name":"john","age":"33","salary":"85000"}|
|10005|sam |30 |75000 |{"id":"10005","name":"sam","age":"30","salary":"75000"}|
+-----+----+---+------+----------------------------------------------------+

简单拆解下这段代码的逻辑：

• struct([df[x] for x in df.columns])：把DataFrame的所有列打包成一个结构体（struct）对象，相当于把每一行的所有字段整合到一个嵌套结构里
• to_json()：将这个结构体对象序列化为JSON格式的字符串，最终生成我们需要的jsonCol列

内容的提问来源于stack exchange，提问作者vishnu ram