Spark/Scala：对含数组类型列的DataFrame进行组件操作（附示例）

Let's start with your sample DataFrame for reference first:

val df = Seq(
  (1, "CS", 0, (0.1, 0.2, 0.4, 0.5)),
  (4, "Ed", 0, (0.4, 0.8, 0.3, 0.6)),
  (7, "CS", 0, (0.2, 0.5, 0.4, 0.7)),
  (101, "CS", 1, (0.5, 0.7, 0.3, 0.8)),
  (5, "CS", 1, (0.4, 0.2, 0.6, 0.9))
).toDF("id", "dept", "test", "array")

Below are some practical operations you might want to run on the array column, with step-by-step code and explanations:

1. Split Array into Separate Columns #

If you want to extract each element of the array into its own dedicated column (your array has 4 elements, so we'll create 4 new columns):

import org.apache.spark.sql.functions._

val splitArrayDF = df
  .withColumn("elem_1", col("array").getItem(0))
  .withColumn("elem_2", col("array").getItem(1))
  .withColumn("elem_3", col("array").getItem(2))
  .withColumn("elem_4", col("array").getItem(3))

splitArrayDF.show()

This will add four new columns, each holding the corresponding value from the original array column.

2. Calculate Statistical Metrics for Array Elements #

You can compute sum, average, max, or min values across the array elements:

val arrayStatsDF = df
  .withColumn("array_sum", aggregate(col("array"), lit(0.0), (acc, x) => acc + x))
  .withColumn("array_avg", avg(col("array")))
  .withColumn("array_max", array_max(col("array")))
  .withColumn("array_min", array_min(col("array")))

arrayStatsDF.show()

• aggregate is used here to sum elements (Spark's default sum function doesn't work directly on array columns).
• array_max/array_min are built-in functions to grab the highest/lowest values in the array.

3. Filter Rows Based on Array Conditions #

Suppose you want to keep only rows where the array has at least one element greater than 0.7:

val filteredDF = df.filter(array_exists(col("array"), x => x > 0.7))

filteredDF.show()

Or if you want rows where all elements are less than 0.8:

val filteredAllDF = df.filter(array_forall(col("array"), x => x < 0.8))

filteredAllDF.show()

• array_exists checks if any element meets the condition.
• array_forall verifies that every element meets the condition.

4. Explode Array and Aggregate by Department + Element #

To break down the array into individual rows (one per element) and then count occurrences grouped by department:

val explodedDF = df.select(col("dept"), explode(col("array")).as("elem"))
  .groupBy("dept", "elem")
  .count()

explodedDF.show()

explode transforms each array element into a separate row, making it easy to aggregate at the element level.

5. Sort Array Elements #

To sort the array in ascending or descending order:

val sortedArrayDF = df
  .withColumn("sorted_asc", array_sort(col("array")))
  .withColumn("sorted_desc", reverse(array_sort(col("array"))))

sortedArrayDF.show()

• array_sort sorts the array in ascending order by default.
• Wrap it with reverse to get a descending-sorted array.

内容的提问来源于stack exchange，提问作者Guanghua Shu