如何在SciPy中对含NaN的多组DataFrame执行单因素ANOVA分析

问题背景 #

调用SciPy的f_oneway执行单因素ANOVA分析，现有3组对应不同分组的DataFrame，需沿axis=1维度计算。初始运行后输出的p值数组包含大量NaN，推测是数据中存在NaN所致。手动剔除每组数据中的NaN后，触发维度不匹配的ValueError：

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (10,) + inhomogeneous part.

需要一种可替代方法，在SciPy中实现与原函数性能一致的ANOVA分析，同时自动忽略原始样本中的NaN值。

解决方案 #

核心问题是f_oneway在指定axis=1时无法处理每行有效数据长度不一致的情况。正确的做法是逐行单独处理：对每行的3组数据分别剔除NaN后，检查每组是否满足ANOVA的最小样本要求（至少2个有效数据），再执行分析。

可行代码 #

from scipy.stats import f_oneway
import pandas as pd
import numpy as np

# 原始数据定义
group1 = {'1': {0: 574145.477641226, 1: 1570531.589742876, 2: 787929.7027375237, 3: 2570860.248729332, 4: 161008.90274193016, 5: np.nan, 6: 1027027.5447738492, 7: 10620164.126712576, 8: 3030551.86415567, 9: 6080226.794887304}, '2': {0: 5590292.274747584, 1: 2015192.4244239724, 2: 1442638.778579319, 3: 9484756.854645137, 4: 231213.53284854395, 5: 1576095.5497571388, 6: 853517.4230997175, 7: 13701076.997994969, 8: 880909.9414626792,9: 10973682.322579961}, '3': {0: 1786259.070812378, 1: 1188813.4685229606, 2: 280628.96027922264, 3: 2752454.6157454816, 4: 142423.39853381264, 5: 408643.1442709076, 6: 978859.742220046, 7: 8581569.49299859, 8: 2810091.19540494, 9: 3250847.2113601067}, '4': {0: 1423158.826220004, 1: np.nan, 2: 659142.6504867233, 3: 2727740.4095105752, 4: np.nan, 5: np.nan, 6: 166867.88656477776, 7: 15578367.076207979, 8: 1262229.6767083204, 9: 7537134.164088669}}

group2 = {'1': {0: 1108031.2785915325, 1: 39475.12143335618, 2: 124744.55052420696, 3: 3415955.3418994714, 4: np.nan, 5: np.nan, 6: 1185929.1264358065, 7: 14219856.696859175, 8: 107938.85576451271, 9: 9075885.57144011}, '2': {0: 3711668.7595074927, 1: np.nan, 2: 92069.12449997541, 3: 1430920.365911842, 4: 23305.980330372353, 5: 146884.88381736717, 6: 143162.52169470832, 7: 11043912.321755221,8: 1507299.549731886, 9: 6675740.20722453}, '3': {0: np.nan, 1: np.nan, 2: np.nan, 3: 192966.31343644214, 4: np.nan, 5: np.nan, 6: np.nan, 7: 13478434.128944362, 8: np.nan, 9: np.nan}, '4': {0: 6446934.0065947445, 1: 3195385.066201132, 2: 3332326.9653299027, 3: 7082529.01041953, 4: 139891.94206563127, 5: 208662.14176584402, 6: 2559284.7669506934, 7: 7395774.107780765, 8: 415796.834504837, 9: 9502289.070542539}}
group3 = {'1': {0: 5832002.081448822, 1: 2607987.6485992945, 2: 2465656.4470221293, 3: 6077038.510021252, 4: 391523.2907555177, 5: np.nan, 6: 2590061.00923242, 7: 7982067.848957288, 8: 61836.18519446156, 9: 10673885.385156194}, '2': {0: 4515593.798793708, 1: 2070893.600738691, 2: 1788619.7598766778, 3: 7302148.61285157, 4: 132247.07494014164, 5: 2130531.009443398, 6: 849079.4122880008, 7: 11086507.936560597, 8: np.nan, 9: 8977041.57285477}, '3': {0: 6916739.909404968, 1: 2886026.824106484, 2: 871822.3682870092, 3: 6515743.347648245, 4: 347767.01169986156, 5: 2975827.5336636542, 6: 3270053.676901515, 7: 9230036.81889698, 8: 4753111.521553177, 9: 11835765.28309747}, '4': {0: 8918243.297089897, 1: 2631751.3775385492, 2: 2294251.0955892503, 3: 7540353.19469351, 4: 48925.64795198818, 5: 447721.0646689915, 6: 1682494.645617865, 7: 6945276.49780706, 8: 978022.2657575278, 9: 11631856.25162219}}

# 转换为DataFrame
df1 = pd.DataFrame(group1)
df2 = pd.DataFrame(group2)
df3 = pd.DataFrame(group3)

# 初始化p值数组
p_values = np.full(len(df1), np.nan)

# 逐行处理ANOVA
for idx in range(len(df1)):
    # 提取每行数据并剔除NaN
    g1 = df1.iloc[idx].dropna().values
    g2 = df2.iloc[idx].dropna().values
    g3 = df3.iloc[idx].dropna().values
    
    # 确保每个组至少有2个有效数据（ANOVA的最低要求）
    if len(g1) >= 2 and len(g2) >= 2 and len(g3) >= 2:
        stat, p = f_oneway(g1, g2, g3)
        p_values[idx] = p

print("p值数组:", p_values)

方案说明 #

• 逐行处理：避免了维度不匹配问题，每行单独计算，每组的有效数据长度无需统一
• 自动过滤无效行：仅当3组都满足至少2个有效样本时才执行ANOVA，否则保留NaN，与原函数行为一致
• 结果一致性：有效行的p值与原代码输出一致，同时能计算出原代码中因NaN被屏蔽的其他行的有效p值

内容的提问来源于stack exchange，提问作者Guilherme Reis