Python新手求助：基于groupby按条件删除数据集行

Hey Marion, let's tackle this step by step—sounds like you're already halfway there with the Counter() part, great job! For your 30k+ row dataset, pandas is the perfect tool to avoid manual indexing and handle both requirements efficiently. Here's how to wrap up the second requirement and even streamline the first step:

Step 1: Remove Demands That Appear Only Once #

First, let's replace the Counter() approach with a more pandas-native method optimized for large datasets. This will filter out any rows where demand occurs just once:

import pandas as pd

# Load your dataset (adjust the file path/format as needed)
df = pd.read_csv("your_dataset.csv")

# Get demands that appear more than once
valid_demands = df["demand"].value_counts()[df["demand"].value_counts() > 1].index
# Filter the dataset to keep only these demands
df = df[df["demand"].isin(valid_demands)]

If you prefer sticking with Counter(), that works too—but pandas' value_counts() is faster for big datasets and integrates seamlessly with the rest of the workflow.

Step 2: Remove Demands With Only One Unique id_user #

Next, we need to drop any remaining demand groups that only have a single unique id_user. Using pandas' grouping and transform() method lets us compute this without looping through rows manually:

# Calculate the number of unique users per demand, and add it as a new column to every row in the group
df["unique_user_count"] = df.groupby("demand")["id_user"].transform("nunique")

# Filter to keep only rows where the demand has more than one unique user
df_final = df[df["unique_user_count"] > 1].drop(columns=["unique_user_count"])

The transform() function is key here—it takes the group-level calculation (unique users per demand) and applies it to every row in that group, so we can easily filter the entire dataset in one go.

Bonus: Combine Both Steps for Efficiency #

If you want to do everything in one pass to save computation time (great for large datasets), you can calculate both metrics at once:

# Add columns for demand occurrence count and unique user count
df["demand_occurrences"] = df.groupby("demand")["demand"].transform("count")
df["unique_user_count"] = df.groupby("demand")["id_user"].transform("nunique")

# Apply both filters in a single step
df_final = df[(df["demand_occurrences"] > 1) & (df["unique_user_count"] > 1)].drop(columns=["demand_occurrences", "unique_user_count"])

This way, you only process the dataset a couple of times instead of multiple separate filtering steps, which is much faster for 30k+ rows.

Just make sure to replace "your_dataset.csv" with your actual file path, and adjust column names if they don't match your dataset. Let me know if you run into any snags!

内容的提问来源于stack exchange，提问作者Marion