PySpark：将DataFrame中所有-1值替换为null的最简方法

The most concise and efficient way to handle this is to iterate over all columns in your DataFrame and use when() with isin() to target both string "-1" and numeric -1 values, replacing them with None (which translates to null in PySpark). This approach works for both string and numeric columns without needing separate logic for each type.

Here's the code:

from pyspark.sql.functions import col, when

# Replace all "-1" (string) and -1 (numeric) with null
processed_df = pyspark_df.select(
    *[when(col(c).isin("-1", -1), None).otherwise(col(c)).alias(c) for c in pyspark_df.columns]
)

# Show the result
processed_df.show()

Output: #

+------+------+-----+
|Number|Letter|Value|
+------+------+-----+
|     1|     A|   30|
|     2|  null|   30|
|  null|     B|   30|
|  null|     A|  null|
+------+------+-----+

How it works: #

• isin("-1", -1) checks if the column value matches either the string "-1" or numeric -1. For string columns, the numeric -1 comparison will return false, so only "-1" gets replaced. For numeric columns, the string "-1" comparison returns false, so only -1 gets replaced.
• when(..., None) replaces matching values with None, which PySpark converts to null.
• The list comprehension *[...] applies this logic to every column in the DataFrame, keeping the original column names with alias(c).

This method is clean, scalable (works no matter how many columns you have), and avoids writing repetitive code for each column.

内容的提问来源于stack exchange，提问作者sanjayr