Spark DataFrame生成唯一标识符：解决10万+行哈希重复问题

First, let's fix a common oversight in your original code: when concatenating columns, always use a delimiter that won't appear in your column values to avoid accidental duplicate strings. For example, if you have a Type value like "Vegtom" and Item like "ato", concatenating without a delimiter would produce the same string as Type "Veg" and Item "tomato". Let's address that first with concat_ws:

val df = Seq( ("Veg", "tomato", 1.99), ("Veg", "potato", 0.45), ("Fruit", "apple", 0.99), ("Fruit", "pineapple", 2.59) ).toDF("Type", "Item", "Price")
df.withColumn("concatenated", concat_ws("|", $"Type", $"Item", $"Price"))

Now, let's go through reliable methods to generate truly unique IDs for your large dataset:

1. Use a Cryptographically Stronger Hash Function #

MD5 is known to have non-negligible collision risks at scale. Switch to SHA-256 or SHA-512—these functions have collision probabilities so low they're effectively zero for most real-world use cases. Spark's built-in sha2 function supports this:

import org.apache.spark.sql.functions.{sha2, concat_ws}

df.withColumn("unique_id", sha2(concat_ws("|", $"Type", $"Item", $"Price"), 256))
  .show(false)

For even lower risk, replace 256 with 512 to generate a longer hash string.

2. Add a Monotonically Increasing ID #

Spark provides monotonically_increasing_id() which generates a unique 64-bit integer for every row. This is guaranteed to be unique across your entire dataset (it's not consecutive, but that's irrelevant for uniqueness):

import org.apache.spark.sql.functions.monotonically_increasing_id

df.withColumn("unique_id", monotonically_increasing_id())
  .show(false)

If you need a human-readable string format, convert the integer to a hex string with hex():

import org.apache.spark.sql.functions.{monotonically_increasing_id, hex}

df.withColumn("unique_id", hex(monotonically_increasing_id()))
  .show(false)

3. Combine Hash + Auto-Increment ID (Dual Guarantee) #

For absolute safety against both hash collisions and accidental duplicate concatenated strings, combine a strong hash with an auto-increment ID. This creates a failsafe identifier:

import org.apache.spark.sql.functions.{sha2, concat_ws, monotonically_increasing_id, concat}

df.withColumn("hash_part", sha2(concat_ws("|", $"Type", $"Item", $"Price"), 256))
  .withColumn("unique_id", concat($"hash_part", "_", monotonically_increasing_id()))
  .show(false)

Even if two rows somehow produce the same hash (extremely unlikely), the suffix integer will differentiate them.

4. Generate Deterministic UUIDs #

If you prefer UUID-format identifiers, create a deterministic UUID based on your concatenated columns using a UDF. This uses nameUUIDFromBytes, which generates the same UUID for identical input strings (great for reproducibility):

import java.util.UUID
import org.apache.spark.sql.functions.{concat_ws, udf}

val generateUUID = udf((input: String) => UUID.nameUUIDFromBytes(input.getBytes).toString)

df.withColumn("concatenated", concat_ws("|", $"Type", $"Item", $"Price"))
  .withColumn("unique_id", generateUUID($"concatenated"))
  .show(false)

Unlike random UUIDs, this method ensures consistent IDs for the same row data across runs.

内容的提问来源于stack exchange，提问作者Akki