含inf值数据的Winsorize处理问题：inf值未被替换的解决方法

给定如下数据集：

{'Date': {0: Timestamp('2016-05-10 00:00:00'), 1: Timestamp('2016-05-11 00:00:00'), 2: Timestamp('2016-05-12 00:00:00'), 3: Timestamp('2016-05-13 00:00:00'), 4: Timestamp('2016-05-14 00:00:00'), 5: Timestamp('2016-05-15 00:00:00'), 6: Timestamp('2016-05-16 00:00:00'), 7: Timestamp('2016-05-17 00:00:00'), 8: Timestamp('2016-05-18 00:00:00'), 9: Timestamp('2016-05-19 00:00:00'), 10: Timestamp('2016-05-20 00:00:00'), 11: Timestamp('2016-05-21 00:00:00')}, 'variable': {0: 0.8491095654077778, 1: 0.40413794873666165, 2: 0.21376796260531072, 3: 0.07286348591835323, 4: 0.5721419081310215, 5: 0.35390011440489355, 6: 0.010530093815129149, 7: 0.013325613826142724, 8: 0.9207323422142153, 9: inf, 10: 0.5726367316365334, 11: 0.8351572257132013}}

尝试用以下代码对variable列执行Winsorize缩尾处理：

df_inf['variable'] = winsorize(df_inf['variable'], limits=[0.05, 0.05], inclusive=(True, True), inplace=False, axis=None, nan_policy='omit')

但数据中的inf值并未被替换，需要让Winsorize处理覆盖inf值。

方法1：先将inf转换为NaN，再执行缩尾处理 #

scipy.stats.mstats.winsorize默认会忽略NaN，但不会识别处理inf，所以第一步先把inf转为NaN，再进行缩尾：

import numpy as np
from scipy.stats.mstats import winsorize

# 替换所有正负inf为NaN
df_inf['variable'] = df_inf['variable'].replace([np.inf, -np.inf], np.nan)
# 执行Winsorize缩尾，nan_policy='omit'会忽略NaN计算分位数
df_inf['variable'] = winsorize(df_inf['variable'], limits=[0.05, 0.05], inclusive=(True, True), nan_policy='omit')

如果不想修改原始数据，可以先创建DataFrame副本再操作。

方法2：手动计算阈值，精准替换inf #

先基于有效数值计算Winsorize的上下限，再把inf替换为上限值，最后执行缩尾：

import numpy as np
from scipy.stats.mstats import winsorize

# 筛选出排除inf的有效数值
valid_values = df_inf['variable'][~df_inf['variable'].isin([np.inf, -np.inf])]
# 计算5%和95%分位数作为缩尾阈值
lower_thresh = np.percentile(valid_values, 5)
upper_thresh = np.percentile(valid_values, 95)
# 替换inf为上限阈值，再执行Winsorize
df_inf['variable'] = winsorize(
    df_inf['variable'].replace(np.inf, upper_thresh),
    limits=[0.05, 0.05],
    inclusive=(True, True)
)

这种方法保证inf被替换为和缩尾上限一致的值，避免出现偏差。

关键注意点 #

• 确保使用scipy.stats.mstats.winsorize而非scipy.stats.winsorize，前者支持处理包含NaN的数组。
• nan_policy='omit'参数会在计算分位数时跳过NaN，不影响阈值的准确性。

内容的提问来源于stack exchange，提问作者newcomer