如何从DataFrame中移除曾出现异常极大值的传感器数据

Here's a straightforward approach to filter out all data from sensors that have ever produced an extreme "infinity-like" value (e.g., 9.9e+37):

Step 1: Identify Faulty Sensors #

First, we detect which sensors have at least one extreme value in their Response column. We check for values matching the known threshold (9.9e+37) and collect the unique sensor IDs that meet this condition.

Step 2: Filter the DataFrame #

Once we have the list of faulty sensors, we exclude all rows belonging to those sensors from the original DataFrame.

Complete Code Example #

import pandas as pd

# Sample data matching your input
data = {
    'Scan': [1,2,3,4,5,6,3,4,5,6,4,5,6],
    'Zeit': ['04.09.2019 06:28:22:405', '04.09.2019 06:28:32:389', '04.09.2019 06:28:42:389', 
             '04.09.2019 06:28:52:389', '04.09.2019 06:29:02:389', '04.09.2019 06:29:12:389',
             '04.09.2019 06:28:42:389', '04.09.2019 06:28:52:389', '04.09.2019 06:29:02:389',
             '04.09.2019 06:29:12:389', '04.09.2019 06:28:52:389', '04.09.2019 06:29:02:389',
             '04.09.2019 06:29:12:389'],
    'Sensor': [101,101,101,101,101,101,102,102,102,102,103,103,103],
    'Response': [9936.3,9958.0,9958.0,9979.7,9979.7,9936.3,9958.0,9.9e+37,9.9e+37,9936.3,7563.5,9871.1,10354.8]
}

df_raw = pd.DataFrame(data)

# Define the extreme value threshold
extreme_threshold = 9.9e+37

# Get unique sensors that have at least one extreme value
faulty_sensors = df_raw[df_raw['Response'] == extreme_threshold]['Sensor'].unique()

# Filter out all rows from faulty sensors
df_clean = df_raw[~df_raw['Sensor'].isin(faulty_sensors)]

# Verify the result
print("Cleaned DataFrame:")
print(df_clean)

Explanation #

• df_raw[df_raw['Response'] == extreme_threshold]: Selects all rows where the response matches the extreme value.
• ['Sensor'].unique(): Extracts unique sensor IDs from those rows, giving us the list of faulty sensors.
• ~df_raw['Sensor'].isin(faulty_sensors): Creates a boolean mask where True indicates rows not from faulty sensors. Applying this mask filters the DataFrame to keep only valid sensor data.

Output #

The cleaned DataFrame will exclude all rows from sensor 102 (since it had extreme values), retaining data from sensors 101 and 103.

内容的提问来源于stack exchange，提问作者Ben