Pandas合并DataFrame获取仅存在于df的非重复记录问题

Your current merge code pulls in columns from both df and df_existing, which is why you're seeing those _x/_y suffixes. The goal is to keep only the rows from df that don't exist in df_existing (matching on symbolid, timeframeid, datetime) while preserving df's original column setup. Here are two simple ways to do this:

Method 1: Adjust Your Existing Merge #

You can modify your merge code to select only the original columns from df after filtering:

merged = df.merge(df_existing,
                  on=['symbolid', 'timeframeid', 'datetime'],
                  how='left',
                  indicator=True)

# Keep only rows unique to df, then select the original columns
df2 = merged[merged['_merge'] == 'left_only'][df.columns]

This works because the merged DataFrame still contains all original columns from df. By specifying df.columns, we ignore any columns that came from df_existing.

Method 2: Index-Based Filtering (Faster for Big Data) #

If you're working with large datasets, this approach avoids merging unnecessary columns entirely:

# Create composite indexes using the key columns
df_keys = df.set_index(['symbolid', 'timeframeid', 'datetime']).index
existing_keys = df_existing.set_index(['symbolid', 'timeframeid', 'datetime']).index

# Filter out rows in df that exist in df_existing
df2 = df[~df_keys.isin(existing_keys)]

This checks if each row's key combination in df is present in df_existing, then keeps only the rows that aren't.

Either method will give you the exact structure of df with only the non-duplicate rows you need.

内容的提问来源于stack exchange，提问作者Stein Erik Hansen