如何在Pandas DataFrame中使用自定义均值计算标准差？

需求说明 #

按X和Y分组计算标准差，但标准差公式中需使用整个X组的全局均值，而非X+Y子分组的均值。

示例数据集 #

import pandas as pd
import numpy as np

data = {
    'X': ['asdf'] * 15,
    'Y': [1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3],
    'A': [58781, 60775, 61424, 61620, 60882, 58788, 57939, 60212, 59086, 59119, 59119, 59119, 59119, 59119, 59119],
    'B': [1.2, 1.6, 1.7, 2.1, 2.3, 2.8, 2.2, 1.9, 2.3, 2.2, 2.2, 2, 2.3, 2.4, 2.5],
    'C': [4.4, 4.2, 5.8, 4, 4.3, 4.5, 4.3, 5.2, 5, 3.8, 4.1, 4.5, 4.4, 4.5, 4.3]
}

df = pd.DataFrame(data)

原代码问题分析 #

原代码通过groupby.apply实现，但df[features] - x_means返回NaN，最终得到全0的标准差结果：

def custom_std(df, means, features, axis=0, ddof=0):
    x = df['X'].iloc[0]
    x_means = pd.concat([means[means['X'] == x][f]] * len(df[features]), ignore_index=True)
    sum_diff_sqr = np.sum(np.square(df[features] - x_means), axis=axis)
    variance = sum_diff_sqr/(len(df[features]) - ddof)
    std_dev = np.sqrt(variance)
    return std_dev

df_means = df.groupby(['X'])['A', 'B', 'C'].mean(numeric_only=True).reset_index()
df_custom_std = df.groupby(['X', 'Y']).apply(custom_std, df_means, ['A', 'B', 'C']).reset_index()

核心问题：用concat重复单行均值生成的DataFrame，与分组数据的列索引/结构不匹配，导致减法运算出现NaN。

解决方案 #

方法1：合并全局均值后计算（推荐） #

先将每个X对应的全局均值合并到原数据集，再按X+Y分组计算，逻辑更清晰：

# 计算每个X的全局均值，添加后缀区分
df_global_means = df.groupby('X')[['A', 'B', 'C']].mean(numeric_only=True).add_suffix('_global_mean')

# 合并均值到原数据
df_merged = df.merge(df_global_means, on='X', how='left')

# 自定义标准差计算函数
def calc_custom_std(group, features):
    result = {}
    for feat in features:
        # 用全局均值计算离均差平方和
        diff = group[feat] - group[f'{feat}_global_mean']
        variance = (diff ** 2).sum() / (len(group) - 0)  # ddof=0，按需调整为1
        result[feat] = np.sqrt(variance)
    return pd.Series(result)

# 按X+Y分组计算
final_result = df_merged.groupby(['X', 'Y']).apply(calc_custom_std, ['A', 'B', 'C']).reset_index()
print(final_result)

方法2：优化apply中的均值匹配 #

直接提取X对应的均值Series，通过广播匹配分组数据长度，避免结构不匹配：

def custom_std_optimized(group, global_means, features):
    x_val = group['X'].iloc[0]
    # 提取当前X的全局均值，转为Series
    mean_vals = global_means.loc[global_means['X'] == x_val, features].iloc[0]
    # 逐个特征计算标准差
    std_values = []
    for feat in features:
        diff = group[feat] - mean_vals[feat]
        var = (diff ** 2).sum() / (len(group) - 0)
        std_values.append(np.sqrt(var))
    return pd.Series(std_values, index=features)

df_means = df.groupby(['X'])['A', 'B', 'C'].mean(numeric_only=True).reset_index()
final_result = df.groupby(['X', 'Y']).apply(custom_std_optimized, df_means, ['A', 'B', 'C']).reset_index()
print(final_result)

关键注意点 #

• ddof参数：计算总体标准差用ddof=0，样本标准差用ddof=1，按需调整
• 维度匹配：确保均值与分组数据的维度一致，避免因结构不匹配产生NaN
• 性能优先：数据量较大时，优先选择合并均值的方法，比apply循环更高效

内容的提问来源于stack exchange，提问作者luarpy