Spark 1.6与Scala 2.10中DataFrame字符串列转整数求和方法

Hey there! The core problem here is your count column is stored as a string—trying to sum it directly would either give wrong results or throw errors. We need to convert it to an integer type first, and here are two straightforward ways to do this:

Method 1: Convert the column upfront, then run your aggregation #

This approach modifies the count column to integer type first, making all subsequent operations work as expected:

// Import the desc function for sorting (required in Spark 1.6)
import org.apache.spark.sql.functions.desc

// Convert the string "count" column to integer type
val dfWithIntCount = df.withColumn("count", df("count").cast("int"))

// Execute your original aggregation logic on the updated DataFrame
val topDestinations = dfWithIntCount.groupBy("DEST_COUNTRY_NAME")
  .sum("count")
  .withColumnRenamed("sum(count)", "destination_total")
  .sort(desc("destination_total"))
  .limit(5)
  .collect()

Method 2: Cast directly within the sum operation #

If you want to keep the original DataFrame unchanged, you can cast the column right inside the sum call:

import org.apache.spark.sql.functions.desc

val topDestinations = df.groupBy("DEST_COUNTRY_NAME")
  .sum(df("count").cast("int").alias("count")) // Cast to integer before summing
  .withColumnRenamed("sum(count)", "destination_total")
  .sort(desc("destination_total"))
  .limit(5)
  .collect()

Key Notes: #

• cast("int") is a shorthand; you can also use cast(IntegerType) if you prefer explicit type objects (just import org.apache.spark.sql.types.IntegerType first).
• Skip using collect(5)—the collect() method doesn't accept parameters. Since you've already used limit(5) to restrict results to 5 rows, collect() will return exactly those 5 rows as an array of Row objects.
• Converting to integer ensures sum calculates a numeric total, instead of trying to concatenate strings (which is the default behavior for string columns).

内容的提问来源于stack exchange，提问作者Carlota