基于PySpark DataFrame三列优先级生成新列的实现方法

构造测试数据 #

先还原你的测试DataFrame，方便复现验证：

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when, coalesce, trim

spark = SparkSession.builder.appName("PriorityColumn").getOrCreate()

data = [
    ("Ginder", "Ginder v. Ginder", "Carrie Diane GINDER v. Carl L. GINDER"),
    ("KENNEY", "KENNEY v. BARNHART", ""),
    ("", "United States v. $933,075.00 IN UNITED STATES CURRENCY", ""),
    ("", "", ""),
    ("Rieser", "", "")
]

df = spark.createDataFrame(data, ["short_1", "short_2", "full_nm"])

两种实现方案 #

方案1：when/otherwise链式判断（完全匹配伪逻辑）

直接按照你给出的优先级逻辑，用when依次判断列是否非空，匹配到第一个符合条件的列就取值，否则返回空字符串：

df_with_new = df.withColumn(
    "new",
    when(col("full_nm") != "", col("full_nm"))
    .when(col("short_2") != "", col("short_2"))
    .when(col("short_1") != "", col("short_1"))
    .otherwise("")
)

方案2：coalesce函数（更简洁）

先将空字符串转为null，利用coalesce取第一个非null值的特性，最后兜底返回空字符串：

df_with_new = df.withColumn(
    "new",
    coalesce(
        when(col("full_nm") != "", col("full_nm")),
        when(col("short_2") != "", col("short_2")),
        when(col("short_1") != "", col("short_1"))
    ).otherwise("")
)

验证结果 #

执行df_with_new.show(truncate=False)，输出与你预期完全一致：

+-------+-------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
|short_1|short_2                                                |full_nm                                                    |new                                                        |
+-------+-------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
|Ginder |Ginder v. Ginder                                       |Carrie Diane GINDER v. Carl L. GINDER                      |Carrie Diane GINDER v. Carl L. GINDER                      |
|KENNEY |KENNEY v. BARNHART                                     |                                                           |KENNEY v. BARNHART                                         |
|       |United States v. $933,075.00 IN UNITED STATES CURRENCY |                                                           |United States v. $933,075.00 IN UNITED STATES CURRENCY     |
|       |                                                       |                                                           |                                                           |
|Rieser |                                                       |                                                           |Rieser                                                     |
+-------+-------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+

注意事项 #

如果数据中存在仅由空格组成的空值（比如" "），可以用trim(col("列名")) != ""替代原判断条件，避免误判为非空值。

内容的提问来源于stack exchange，提问作者emiley mille