Pandas DataFrame如何删除Level列孤立假阳性行且保留首行值

核心思路 #

统一Level列的大小写避免判断误差，首行无条件保留，非首行通过判断当前值是否为前后范围内的孤立值（出现次数≤2）过滤假阳性，提供两种实现方法可选。

方法1：滑动窗口众数法（逻辑简单易调整） #

适合中小数据量，可灵活适配不同的假阳性最大长度：

import pandas as pd

# 构造示例数据，可替换为你自己的DataFrame
data = {'Level': ['Difficult', 'Difficult', 'Difficult', 'Difficult', 'Easy', 'Difficult', 'easy', 'difficult', 'difficult', 'difficult']}
df = pd.DataFrame(data)

# 1. 新增临时列统一为小写，保留原列的大小写格式
df['level_lower'] = df['Level'].str.lower()

# 2. 单独保留首行
first_row = df.iloc[[0]]

# 3. 处理剩余行：5位居中滑动窗口取众数，过滤和众数不一致的孤立假阳性
# 若假阳性最多为N个，窗口大小设置为2*N +1即可，此处N=2所以窗口大小为5
rest_df = df.iloc[1:]
rest_df['window_mode'] = rest_df['level_lower'].rolling(
    window=5, 
    center=True, 
    min_periods=1
).apply(lambda x: x.mode()[0])
filtered_rest = rest_df[rest_df['level_lower'] == rest_df['window_mode']].drop(columns=['level_lower', 'window_mode'])

# 4. 合并结果
result_df = pd.concat([first_row.drop(columns='level_lower'), filtered_rest], ignore_index=True)

方法2：位移判断法（效率更高适合大数据量） #

通过位移获取前后两行的值，直接统计相同值数量判断是否为孤立值，运行速度更快：

import pandas as pd

data = {'Level': ['Difficult', 'Difficult', 'Difficult', 'Difficult', 'Easy', 'Difficult', 'easy', 'difficult', 'difficult', 'difficult']}
df = pd.DataFrame(data)

df['level_lower'] = df['Level'].str.lower()
# 获取前后各两行的值
df['prev1'] = df['level_lower'].shift(1)
df['prev2'] = df['level_lower'].shift(2)
df['next1'] = df['level_lower'].shift(-1)
df['next2'] = df['level_lower'].shift(-2)

# 首行保留，非首行前后相同值不足2个判定为假阳性过滤
keep_list = []
for idx, row in df.iterrows():
    if idx == 0:
        keep_list.append(True)
        continue
    same_cnt = sum([
        row['prev1'] == row['level_lower'],
        row['prev2'] == row['level_lower'],
        row['next1'] == row['level_lower'],
        row['next2'] == row['level_lower']
    ])
    keep_list.append(same_cnt >= 2)

result_df = df[keep_list].drop(columns=['level_lower', 'prev1', 'prev2', 'next1', 'next2'])

参数调整说明 #

• 若假阳性的最大长度不是2而是1，可将方法1的窗口大小改为3，方法2的判断阈值改为same_cnt >=1即可
• 不需要保留原大小写的话，可直接修改原Level列为小写，省去临时列步骤

内容的提问来源于stack exchange，提问作者Elgun Valiyev