如何在保留国家信息的前提下将分组姓名转换为独立人员条目

你可以通过PySpark的split和explode函数组合实现需求，具体步骤如下：

1. 导入必要函数 #

from pyspark.sql import SparkSession
from pyspark.sql.functions import split, explode, col

2. 创建示例DataFrame（模拟你的输入数据） #

# 初始化SparkSession
spark = SparkSession.builder.appName("split_and_explode").getOrCreate()

# 构造数据
data = [
    ("jan,marek", "Poland"),
    ("anna,kasia", "Poland"),
    ("john,emma", "New Zealand"),
    ("oliver,ava", "New Zealand"),
    ("tomasz,ewa", "Poland"),
    ("liam,amelia", "New Zealand")
]

df = spark.createDataFrame(data, ["customer_names", "country"])

3. 执行拆分与展开操作 #

# 拆分字符串为数组，再展开数组为多行，并重命名列
result_df = df.withColumn("cx_name", explode(split(col("customer_names"), ","))) \
              .drop("customer_names")

# 显示结果
result_df.show()

输出结果 #

+-----------+-------+
|    country|cx_name|
+-----------+-------+
|     Poland|    jan|
|     Poland|  marek|
|     Poland|   anna|
|     Poland|  kasia|
|New Zealand|   john|
|New Zealand|   emma|
|New Zealand| oliver|
|New Zealand|    ava|
|     Poland| tomasz|
|     Poland|    ewa|
|New Zealand|   liam|
|New Zealand| amelia|
+-----------+-------+

关键步骤说明 #

• split(col("customer_names"), ",")：将customer_names列的逗号分隔字符串拆分为字符串数组
• explode(...)：将数组中的每个元素生成单独的一行，同时保留country列的对应值
• drop("customer_names")：移除不再需要的原列，也可以通过withColumnRenamed直接重命名展开后的列

内容的提问来源于stack exchange，提问作者james milwaukee