Scala Spark如何从单数据列拆分生成多个新数据列

方法1：固定键场景（已知仅包含name、address两个键） #

直接通过字符串拆分函数按规则提取值，逻辑简单高效：

PySpark 示例代码 #

from pyspark.sql import functions as F

df = df.withColumn("name", F.split(F.split(F.col("col_b"), ", ")[0], "=")[1]) \
       .withColumn("address", F.split(F.split(F.col("col_b"), ", ")[1], "=")[1]) \
       .drop("col_b")

Scala 示例代码 #

import org.apache.spark.sql.functions._

val resDf = df.withColumn("name", split(split(col("col_b"), ", ")(0), "=")(1))
              .withColumn("address", split(split(col("col_b"), ", ")(1), "=")(1))
              .drop("col_b")

方法2：通用场景（键数量不固定/后续可能新增键） #

先将col_b转为Map结构再按键取值，扩展性更强：
str_to_map 函数要求Spark版本在2.3及以上

PySpark 示例代码 #

from pyspark.sql import functions as F

df = df.withColumn("col_b_map", F.str_to_map(F.col("col_b"), ", ", "=")) \
       .withColumn("name", F.col("col_b_map")["name"]) \
       .withColumn("address", F.col("col_b_map")["address"]) \
       .drop("col_b", "col_b_map")

Scala 示例代码 #

import org.apache.spark.sql.functions._

val resDf = df.withColumn("col_b_map", str_to_map(col("col_b"), ", ", "="))
              .withColumn("name", col("col_b_map")("name"))
              .withColumn("address", col("col_b_map")("address"))
              .drop("col_b", "col_b_map")

内容的提问来源于stack exchange，提问作者thunderbolt