Pandas groupby.diff()未返回预期输出问题排查与解决

Great question! Let's clarify what's happening here and how to get the nested structure you're expecting.

Why diff() Doesn't Return a Nested Structure (Unlike sum()) #

First, the key difference between sum() and diff() in pandas groupby lies in their function categories:

• sum() is an aggregation function: it collapses each group into a single value (the sum of the group), so pandas returns a DataFrame with a multi-index matching your groupby keys (outer, inner) to represent each group's aggregated result.
• diff() is a transformation function: it returns a value for every row in the original DataFrame (with NaN for the first row of each group, since there's no prior value to compute a difference against). It preserves the original row order and index, which is why you see a 10-row result instead of the compact nested structure from sum().

Important: Your Groupby IS Working Correctly #

It might look like outer isn't being considered, but it actually is! Let's verify with your sample data:

• For outer=0, inner=a: values 78 → 68, diff is -10.0
• For outer=1, inner=b: values 78 →22, diff is -56.0
• For outer=0, inner=e: values 2 →39, diff is 37.0

These are all independent calculations per (outer, inner) pair—you're just seeing the results in the original DataFrame's structure instead of a grouped summary.

How to Get the Nested, Grouped Format for diff() #

If you want results in the same nested multi-index format as sum(), here are a few clean approaches:

Approach 1: Compute Diff, Then Reshape to Multi-Index #

First calculate the diff, then drop the NaN rows (since they don't represent a valid difference) and set the group keys as the index:

import pandas as pd
import numpy as np

df = pd.DataFrame({'inner':list('aabbccddee'),'outer':[0,0,1,1,0,0,1,1,0,0], 'value':np.random.randint(0,100,10)})

# Calculate diff per (outer, inner) group
df['diff'] = df.groupby(['outer','inner'])['value'].diff()

# Reshape to nested multi-index format
nested_diff = df.dropna().set_index(['outer','inner'])[['diff']].sort_index()
print(nested_diff)

This will output something like:

diff
outer inner       
0     a      -10.0
      c      -28.0
      e       37.0
1     b      -56.0
      d      -44.0

Approach 2: Use groupby().apply() for Direct Nested Results #

You can use apply() to run diff() on each group and automatically retain the multi-index:

nested_diff = df.groupby(['outer','inner'])['value'].apply(lambda x: x.diff().dropna())
# Convert to DataFrame for cleaner output
nested_diff = nested_diff.to_frame('diff')
print(nested_diff)

This will give you the same nested structure; if you don't need the inner index representing row positions within each group, you can drop it with .reset_index(level=2, drop=True).

Summary #

• diff() is a transformation function, so it preserves the original DataFrame's row structure—your (outer, inner) grouping is working, just not displayed in a compact way.
• To get the nested multi-index format, you can either reshape the diff results after calculation or use apply() to wrap the diff operation.

内容的提问来源于stack exchange，提问作者Gene Burinsky