基于条件掩码处理Pandas DataFrame：将指定值替换为NaN的最优方法

Nice question! When you need to swap all values above a specific threshold (like 100) with NaN in a pandas DataFrame, the best approach is to use vectorized operations—these are far more efficient than loops or row-wise/apply functions, especially with large datasets.

Let's start with your sample data to demonstrate:

import pandas as pd
import numpy as np

# Original DataFrame
df = pd.DataFrame({'a':[1,250,480], 'b':[60,51,101], 'c':[15,689,1]})

1. Use df.mask() (Most Intuitive for This Use Case) #

The mask() method replaces values where a condition is True with a specified value (default is NaN, which is exactly what we need here). This directly aligns with your goal: "replace values > 100 with NaN".

threshold = 100
df_processed = df.mask(df > threshold)

Running this will give you the desired output:

a     b    c
0  1.0  60.0  15.0
1  NaN  51.0  NaN
2  NaN   NaN   1.0

2. Use df.where() (Alternative Vectorized Option) #

where() does the opposite of mask(): it keeps values where the condition is True, and replaces others with NaN. To use it here, we just invert the condition (keep values ≤ 100):

df_processed = df.where(df <= threshold)

This produces the exact same result as mask()—pick whichever reads more naturally to you.

What to Avoid: Slow Element-Wise Methods #

You might see solutions using applymap() or loops, but these are not optimal for performance, especially with large DataFrames:

# Not recommended - slow for big datasets
df_slow = df.applymap(lambda x: np.nan if x > threshold else x)

Vectorized operations like mask() and where() leverage pandas' underlying C-based optimizations, making them orders of magnitude faster than逐element processing.

内容的提问来源于stack exchange，提问作者pabrao