为何Pandas合并速度快？未按索引排序仍高效，排序优化无提速？

Great question—this is such a relatable confusion when you’re used to manual data matching (like sorting paper docs to make merging easier) but don’t know how Pandas handles merges under the hood! Let’s break this down:

1. Pandas Uses Hash Joins By Default (Not Manual-Style Matching) #

The key difference here is how Pandas performs merges vs. how we do it manually. When you merge two DataFrames with pd.merge(), Pandas defaults to a hash join algorithm, which works like this:

• First, it takes one of your datasets (usually the smaller one) and computes hash values for every entry in the merge column(s).
• It builds a hash table that maps each hash value to the corresponding rows in that dataset.
• Then, it iterates through the second dataset, computes hash values for its merge column entries, and uses the hash table to instantly look up matching rows.

Hash lookups are nearly constant-time operations (O(1)), so the order of your merge columns doesn’t matter much here. The hash table does all the heavy lifting to avoid the slow, line-by-line checking you’d do manually.

2. When Sorting Would Help (And Why It Didn’t In Your Test) #

Sorting only impacts merge performance if you’re using a sort-merge join (where you sort both datasets first, then use a two-pointer technique to match rows). Pandas doesn’t use this by default unless you explicitly set sort=True in your merge call, or if the dataset is so large that the hash table can’t fit in memory (a rare case for most standard-sized datasets).

In your test, since you didn’t force a sort-merge join, sorting the columns didn’t change the underlying hash join logic—hence the 8-second runtime stayed the same. If you wanted to see a difference, you could:

• Try merging extremely large datasets (millions/billions of rows) where hash table memory becomes a bottleneck.
• Explicitly use pd.merge(df1, df2, on='key', sort=True) and compare sorted vs. unsorted runtimes.

3. Manual Merging vs. Pandas: Apples to Oranges #

When you merge paper docs manually, sorting cuts down on the number of checks you have to do (no more flipping back and forth to find matches). But Pandas’ hash join eliminates that problem entirely by creating an index-like structure (the hash table) that lets it find matches instantly, regardless of order.

Your test results are totally expected—Pandas’ default merge algorithm is optimized to be fast even without sorted keys, which is why you didn’t see a speedup after sorting.

内容的提问来源于stack exchange，提问作者Jeffrey Benjamin Brown