如何在Pandas中基于其他行的存在情况移除冗余行（内存友好方案）

Great question—let’s solve this without expensive cartesian products or self-joins, which is critical for keeping memory usage in check with large datasets. Here’s a step-by-step, optimized solution using pandas:

Core Requirement Breakdown #

We need to keep rows that meet either of these rules:

• The row has C = "*", and it’s part of an (A,B) group that includes this * value; OR
• The (A,B) group doesn’t contain any row with C = "*" (so all rows in the group stay)

Step 1: Recreate the Sample DataFrame #

First, let’s set up your test data to validate our solution:

import pandas as pd

# Sample input DataFrame
df = pd.DataFrame({
    'A': ['foo', 'foo', 'foo', 'foo', 'bar', 'bar'],
    'B': [12, 12, 12, 15, 13, 13],
    'C': ['*', 'z', 'x', 'x', 'z', 'x']
})

Step 2: Memory-Friendly Filtering #

We’ll use groupby + transform to avoid costly joins. This method processes data in groups and maps results back to the original DataFrame without creating redundant rows:

# Add a temporary flag: does this (A,B) group have any row with C='*'?
df['has_star'] = df.groupby(['A', 'B'])['C'].transform(lambda x: '*' in x.values)

# Filter rows based on our rules
filtered_df = df[(df['C'] == '*') | (~df['has_star'])]

# Clean up the temporary flag column
filtered_df = filtered_df.drop(columns=['has_star'])

Step 3: Check the Result #

Running this code gives exactly your target output:

A   B  C
0  foo  12  *
3  foo  15  x
4  bar  13  z
5  bar  13  x

Why This Is Memory-Friendly #

• No cartesian products/self-joins: Group operations avoid creating duplicate rows that bloat memory.
• Transform is efficient: The transform method applies logic to each group and aligns results with the original row indices—using minimal extra memory compared to merging separate group datasets.
• Scalable: For extremely large DataFrames, this approach outperforms self-joins by a wide margin, as pandas optimizes group operations for low overhead.

If your DataFrame is extremely large (too big for memory even with this), you could split it by A values, process each chunk separately, then concatenate the results—but for most cases, the above solution will work perfectly.

内容的提问来源于stack exchange，提问作者Malcolm Tucker