Scala Spark DataFrame多列分层条件过滤实现咨询

Hey there! As someone new to Spark and Scala, I totally get how tricky it can be to translate custom filtering rules into code—let's walk through this step by step so you understand exactly what's happening.

Breakdown of Your Filter Logic #

First, let's restate your rules to make sure we're aligned:

• Priority 1: Keep any row where the 4Wheel column is either Subaru or Toyota (no need to check 2Wheel here).
• Priority 2: If 4Wheel is null or an empty string, keep the row only if 2Wheel is Yamaha or Harley.

Scala/Spark Code Implementation #

Here's the code to make this happen, with explanations for each part so you know why it works:

// Import necessary Spark libraries
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

// Set up your SparkSession (you probably already have this)
val spark = SparkSession.builder().appName("VehicleFilter").getOrCreate()
import spark.implicits._

// Apply the priority filter to your DataFrame (named `df` here)
val filteredDF = df.filter(
  // First condition: Priority 1 matches
  (col("4Wheel").isin("Subaru", "Toyota")) ||
  // Second condition: Priority 2 matches (only when 4Wheel is empty)
  (
    (col("4Wheel").isNull || col("4Wheel") === "") &&
    col("2Wheel").isin("Yamaha", "Harley")
  )
)

// Optional: Check the result with a preview
filteredDF.show()

Key Explanations #

• col("4Wheel").isin("Subaru", "Toyota"): This checks if 4Wheel matches either of the two priority brands—simple and clean for exact value matches.
• (col("4Wheel").isNull || col("4Wheel") === ""): This covers both Spark's native null values and empty strings (since "empty" can sometimes mean either in datasets). If your data only uses null (no empty strings), you can simplify this to just col("4Wheel").isNull.
• The || (OR) between the two main groups ensures we first capture all Priority 1 rows, then add in valid Priority 2 rows. The && (AND) in the second group makes sure we only keep rows where both 4Wheel is empty and 2Wheel meets the requirement.

Quick Tips #

• filter and where are interchangeable in Spark—feel free to use whichever you find more readable.
• Test your filter on a small sample first (like df.sample(0.1).show()) to confirm it's capturing the right rows before running it on your full dataset.

内容的提问来源于stack exchange，提问作者barun