Python移除数据集离群值时触发KeyError: 'Price'异常，求解决方案

问题场景 #

目标是移除数据集中包含离群值的行，原始数据集如下：

data = {'Price': [1, 4, 5, 100],
        'Year': [20, 21, 19, 18],
        'Mileage': [100, 1500, 1654, 2024],
        'EngineV': [2, 3, 5, 4]}

尝试用以下代码移除离群值时，出现KeyError: 'Price'错误：

def remove_outliers(data=data,columns=columns,n_std=3):
    for col in columns:
               
        mean = data[col].mean()
        sd = data[col].std()
        df = data[(data[col] <= mean+(n_std*sd))]
        
    return data

df = pd.DataFrame(data.apply(remove_outliers))

完整报错回溯：

KeyError                                  Traceback (most recent call last)
File C:\Python3\lib\site-packages\pandas\core\indexes\base.py:3621, in Index.get_loc(self, key, method, tolerance)
   3620 try:
-> 3621     return self._engine.get_loc(casted_key)
   3622 except KeyError as err:

File C:\Python3\lib\site-packages\pandas\_libs\index.pyx:136, in pandas._libs.index.IndexEngine.get_loc()

File C:\Python3\lib\site-packages\pandas\_libs\index.pyx:144, in pandas._libs.index.IndexEngine.get_loc()

File pandas\_libs\index_class_helper.pxi:41, in pandas._libs.index.Int64Engine._check_type()

KeyError: 'Price'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Input In [13], in <module>
      8         df = data[(data[col] <= mean+(n_std*sd))]
     10     return data
---> 11 df = pd.DataFrame(data.apply(remove_outliers))
     12 print("New Shape :", df.shape)

File C:\Python3\lib\site-packages\pandas\core\frame.py:8839, in DataFrame.apply(self, func, axis, raw, result_type, args, **kwargs)
   8828 from pandas.core.apply import frame_apply
   8830 op = frame_apply(
   8831     self,
   8832     func=func,
   (...)
   8837     kwargs=kwargs,
   8838 )
-> 8839 return op.apply().__finalize__(self, method="apply")

File C:\Python3\lib\site-packages\pandas\core\apply.py:727, in FrameApply.apply(self)
    724 elif self.raw:
    725     return self.apply_raw()
-> 727 return self.apply_standard()

File C:\Python3\lib\site-packages\pandas\core\apply.py:851, in FrameApply.apply_standard(self)
    850 def apply_standard(self):
-> 851     results, res_index = self.apply_series_generator()
    853     # wrap results
    854     return self.wrap_results(results, res_index)

File C:\Python3\lib\site-packages\pandas\core\apply.py:867, in FrameApply.apply_series_generator(self)
    864 with option_context("mode.chained_assignment", None):
    865     for i, v in enumerate(series_gen):
    866         # ignore SettingWithCopy here in case the user mutates
-> 867         results[i] = self.f(v)
    868         if isinstance(results[i], ABCSeries):
    869             # If we have a view on v, we need to make a copy because
    870             #  series_generator will swap out the underlying data
    871             results[i] = results[i].copy(deep=False)

Input In [13], in remove_outliers(data, columns, n_std)
      3 def remove_outliers(data=data,columns=columns,n_std=3):
      4     for col in columns:
----> 6         mean = data[col].mean()
      7         sd = data[col].std()
      8         df = data[(data[col] <= mean+(n_std*sd))]

File C:\Python3\lib\site-packages\pandas\core\series.py:958, in Series.__getitem__(self, key)
    955     return self._values[key]
    957 elif key_is_scalar:
-> 958     return self._get_value(key)
    960 if is_hashable(key):
    961     # Otherwise index.get_value will raise InvalidIndexError
    962     try:
    963         # For labels that don't resolve as scalars like tuples and frozensets

File C:\Python3\lib\site-packages\pandas\core\series.py:1069, in Series._get_value(self, label, takeable)
   1066     return self._values[label]
   1068 # Similar to Index.get_value, but we do not fall back to positional
-> 1069 loc = self.index.get_loc(label)
   1070 return self.index._get_values_for_loc(self, loc, label)

File C:\Python3\lib\site-packages\pandas\core\indexes\base.py:3623, in Index.get_loc(self, key, method, tolerance)
   3621     return self._engine.get_loc(casted_key)
   3622 except KeyError as err:
-> 3623     raise KeyError(key) from err
   3624 except TypeError:
   3625     # If we have a listlike key, _check_indexing_error will raise
   3626     #  InvalidIndexError. Otherwise we fall through and re-raise
   3627     #  the TypeError.
   3628     self._check_indexing_error(key)

曾尝试以下方式指定columns，均未解决问题：

• columns=data[['Price','Mileage','EngineV','Year']]
• columns=("Price","Mileage","EngineV","Year")
• columns=data.iloc[:,[0,1,2,3]]

问题根源 #

1. 错误使用apply()：DataFrame.apply()默认按列处理，会将每一列作为单独的Series传入remove_outliers函数。此时函数内的data是单个Series，而非完整DataFrame，自然没有'Price'这样的列名索引，触发KeyError。
2. 函数逻辑无效：循环中每次重新赋值df但未更新原数据，最终返回的还是原始数据集，根本没有过滤行。
3. 参数默认值风险：函数定义时用columns=columns作为默认参数，若columns变量未提前定义，会导致参数初始化错误。

解决方案 #

方法1：基于3σ原则手动过滤 #

先将原始字典转为DataFrame，再通过掩码累积需要保留的行：

import pandas as pd

# 原始数据转DataFrame
data = {'Price': [1, 4, 5, 100],
        'Year': [20, 21, 19, 18],
        'Mileage': [100, 1500, 1654, 2024],
        'EngineV': [2, 3, 5, 4]}
df = pd.DataFrame(data)

def remove_outliers(df, columns, n_std=3):
    # 初始化掩码：默认保留所有行
    mask = pd.Series([True]*len(df), index=df.index)
    for col in columns:
        mean_val = df[col].mean()
        std_val = df[col].std()
        # 更新掩码：仅保留当前列在3σ范围内的行
        mask &= (df[col] >= mean_val - n_std*std_val) & (df[col] <= mean_val + n_std*std_val)
    # 返回过滤后的结果
    return df[mask]

# 指定需要检查的列
target_columns = ['Price','Mileage','EngineV','Year']
filtered_df = remove_outliers(df, target_columns)

print("原始数据：")
print(df)
print("\n过滤后数据：")
print(filtered_df)

方法2：用z-score简洁过滤 #

借助scipy.stats计算z分数，快速过滤离群值：

import pandas as pd
import scipy.stats as stats

data = {'Price': [1, 4, 5, 100],
        'Year': [20, 21, 19, 18],
        'Mileage': [100, 1500, 1654, 2024],
        'EngineV': [2, 3, 5, 4]}
df = pd.DataFrame(data)

target_columns = ['Price','Mileage','EngineV','Year']
# 计算指定列的z分数
z_scores = stats.zscore(df[target_columns])
# 保留所有列z分数绝对值小于3的行
filtered_df = df[(abs(z_scores) < 3).all(axis=1)]

print(filtered_df)

说明 #

两种方法均能实现目标：移除任意指定列中存在离群值的行。方法1更灵活，可自定义过滤规则；方法2代码更简洁，适合快速实现3σ原则过滤。

内容的提问来源于stack exchange，提问作者Ariel