PySpark中如何对String数据类型列执行groupBy分组操作？

针对包含整型、字符串类型列的DataFrame，直接使用内置聚合函数即可完成按指定列分组、将同组字符串值收集为列表的需求，无需对字符串列做额外类型转换。

• 核心使用函数：collect_list，作用是按分组顺序收集指定列的值，返回数组类型列
• 如果需要对聚合后的值去重，替换为collect_set即可，该函数返回去重后的无序集合

PySpark 实现示例 #

from pyspark.sql.functions import collect_list

# 替换为你自己的DataFrame即可，以下为样例数据构造逻辑
sample_data = [
    (2020, "Inception"),
    (2018, "The Godfather"),
    (2018, "The Dark Knight"),
    (2015, "12 Angry Men"),
    (2020, "Schindler's List"),
    (2015, "Pulp Fiction"),
    (2018, "Fight Club")
]
df = spark.createDataFrame(sample_data, schema=["Year", "Movie"])

# 分组聚合逻辑
result_df = df.groupBy("Year") \
              .agg(collect_list("Movie").alias("Movie")) \
              .orderBy("Year", ascending=False)

# 打印结果
result_df.show(truncate=False)

Scala 实现示例 #

import org.apache.spark.sql.functions.collect_list

// 替换为你自己的DataFrame即可，以下为样例数据构造逻辑
val sampleData = Seq(
    (2020, "Inception"),
    (2018, "The Godfather"),
    (2018, "The Dark Knight"),
    (2015, "12 Angry Men"),
    (2020, "Schindler's List"),
    (2015, "Pulp Fiction"),
    (2018, "Fight Club")
)
val df = sampleData.toDF("Year", "Movie")

// 分组聚合逻辑
val resultDf = df.groupBy("Year")
                 .agg(collect_list("Movie").alias("Movie"))
                 .orderBy($"Year".desc)

// 打印结果
resultDf.show(false)

运行上述代码后，输出结果完全匹配预期格式：

+----+--------------------------------------------+
|Year|Movie                                       |
+----+--------------------------------------------+
|2020|[Inception, Schindler's List]               |
|2018|[The Godfather, The Dark Knight, Fight Club]|
|2015|[12 Angry Men, Pulp Fiction]                |
+----+--------------------------------------------+

提示：如果需要按多列分组，直接在groupBy方法中传入多个列名即可，例如groupBy("col1", "col2")，字符串列的聚合逻辑不受分组列数量影响。

内容的提问来源于stack exchange，提问作者Lokesh Rathi