Spark DataFrame中如何将字符串类型列转换为BigInt类型？

Hey there! Converting your column2 from string to BigInt in Spark is totally straightforward—here are a few reliable methods tailored to different Spark versions and your needs:

Method 1: Use cast() (Works with all Spark versions) #

This is the standard, universal approach. You can either reference the BigIntType directly or use its string alias "bigint" for brevity.

Scala Example #

import org.apache.spark.sql.types.BigIntType
import org.apache.spark.sql.functions.col

// Assuming your original DataFrame is named originalDF
val convertedDF = originalDF.withColumn("column2", col("column2").cast(BigIntType))

// Shorthand version using string type alias
val convertedDF = originalDF.withColumn("column2", col("column2").cast("bigint"))

Python Example #

from pyspark.sql.types import BigIntType

// Assuming your original DataFrame is named original_df
converted_df = original_df.withColumn("column2", original_df["column2"].cast(BigIntType()))

// Shorthand version using string type alias
converted_df = original_df.withColumn("column2", original_df["column2"].cast("bigint"))

Method 2: Use toBigInt() (Spark 3.0+) #

If you're running Spark 3.0 or later, this dedicated method makes your code more readable and explicit:

Scala Example #

import org.apache.spark.sql.functions.col

val convertedDF = originalDF.withColumn("column2", col("column2").toBigInt)

Python Example #

converted_df = original_df.withColumn("column2", original_df["column2"].toBigInt())

Handle Conversion Failures Gracefully #

If column2 might contain non-numeric values, use try_cast() (Spark 2.3+) to avoid runtime errors—it returns null for invalid entries instead of crashing your job:

Scala Example #

import org.apache.spark.sql.functions.try_cast
import org.apache.spark.sql.types.BigIntType

val convertedDF = originalDF.withColumn("column2", try_cast(col("column2"), BigIntType))

Python Example #

from pyspark.sql.functions import try_cast

converted_df = original_df.withColumn("column2", try_cast(original_df["column2"], "bigint"))

Quick side note: BigIntType in Spark maps to Java's BigInteger, which handles integers larger than the 64-bit limit of LongType—perfect if you're working with extra-large numbers!

内容的提问来源于stack exchange，提问作者Dipanjan Das