如何按多条件为PySpark DataFrame新增output列赋值NW

你的代码有3处核心错误导致结果不符合预期：

• 逻辑运算符优先级错误：PySpark中&（与）的优先级高于|（或），你没有给zone的判断条件加括号，导致逻辑执行顺序完全不符合需求
• dcode排除逻辑错误：你写的dcode != "702s" | dcode != "736s" | dcode != "737s"逻辑永远为真，任意dcode都会满足该条件，正确写法是直接用~isin()判断不在指定列表即可
• 列名大小写不匹配：原始DataFrame的区域列名是小写zone，你代码里写的是大写Zone，会触发列不存在的报错

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
import pandas as pd

# 初始化SparkSession
spark = SparkSession.builder.appName("test").getOrCreate()

# 原始测试数据
a = ['480s','480s','499s','499s','650s','650s','702s','702s','736s','736s','736s','737s','737s']
b = ['North','West','East','North','East','North','North','West','North','South','West','North','West']
pd_df = pd.DataFrame(dict(dcode=a, zone=b))

# 转成PySpark DataFrame
df = spark.createDataFrame(pd_df)

# 新增output列的正确逻辑
df_ = df.withColumn("output", 
            F.when(
                # 条件1：zone为North或West
                (F.col("zone").isin("North", "West")) 
                & 
                # 条件2：dcode不在排除列表中，~为取反运算符
                (~F.col("dcode").isin("702s", "736s", "737s")),
                "NW"
            ).otherwise("")
         )

# 打印结果验证
df_.show()

+-----+-----+------+
|dcode| zone|output|
+-----+-----+------+
| 480s|North|    NW|
| 480s| West|    NW|
| 499s| East|      |
| 499s|North|    NW|
| 650s| East|      |
| 650s|North|    NW|
| 702s|North|      |
| 702s| West|      |
| 736s|North|      |
| 736s|South|      |
| 736s| West|      |
| 737s|North|      |
| 737s| West|      |
+-----+-----+------+

内容的提问来源于stack exchange，提问作者naveen kumar