PySpark LinearRegression的meanSquaredError与SAS结果不符问题问询

将SAS代码迁移至PySpark时，尝试复现proc anova功能，发现PySpark中LinearRegression模型输出的meanSquaredError与SAS结果存在差异：PySpark给出的值为402.8744，而手动通过summary.predictions计算得到的MSE（450.2714）与SAS结果一致。这并非bug，而是两者对MSE的计算方式不同。

示例代码 #

from pyspark.ml.regression import LinearRegression
from pyspark.ml.feature import VectorAssembler, StringIndexer, OneHotEncoder
from pyspark.sql.functions import expr

# df is the sashelp.class data set from sas

# StringIndexer for sex
indexer = StringIndexer(inputCol="sex", outputCol="sex_index")
df_indexed = indexer.fit(df).transform(df)

# OneHotEncoder for sex_index
encoder = OneHotEncoder(inputCol="sex_index", outputCol="sex_ohe", dropLast=True)
df_encoded = encoder.fit(df_indexed).transform(df_indexed)

# VectorAssembler with one-hot encoded sex and weight
assembler = VectorAssembler(inputCols=["sex_ohe"], outputCol="features")
df_features = assembler.transform(df_encoded).select("features", "weight")

# Fit a Linear Regression model
lr = LinearRegression(featuresCol="features", labelCol="weight", fitIntercept=True)
lr_model = lr.fit(df_features)

# Calculate degrees of freedom
n = df_features.count()
k = df_encoded.select("sex_ohe").first()["sex_ohe"].size  # one-hot sex
df_regression = k
df_residual = n - k - 1
df_total = n - 1

# Get the sums of squares
ss = lr_model.summary.predictions\
    .withColumn("mean_pred", expr("mean(weight) over()"))\
    .selectExpr("round(sum(POWER(prediction - weight, 2)), 5) as sse",
                "round(sum(POWER(mean_pred - weight, 2)), 5) as sst",
                "round(sum(POWER(mean_pred - prediction, 2)), 5) as ssr")
sse=ss.toPandas()["sse"].tolist()[0]
sst = ss.toPandas()["sst"].tolist()[0]
ssr = ss.toPandas()["ssr"].tolist()[0]

# Calculate F-statistic
msr = ssr / df_regression
mse = sse / df_residual
f_value = round(msr / mse, 2)

anova_table = {
    "Degrees of Freedom Regression": df_regression,
    "Sums of Squares Regression": ssr,
    "Mean Square Regression": round(msr, 5),
    "Degrees of Freedom Residual": df_residual,
    "Sums of Squares Error": sse,
    "Mean Squared Error": round(mse, 5),
    "Degrees of Freedom Total": df_total,
    "Sums of Squares Total": sst,
    "F-Statistic": f_value
}

print(lr_model.summary.meanSquaredError)
anova_table

运行结果 #

402.8744152046783
{'Degrees of Freedom Regression': 1,
 'Sums of Squares Regression': 1681.12295,
 'Mean Square Regression': 1681.12295,
 'Degrees of Freedom Residual': 17,
 'Sums of Squares Error': 7654.61389,
 'Mean Squared Error': 450.27141,
 'Degrees of Freedom Total': 18,
 'Sums of Squares Total': 9335.73684,
 'F-Statistic': 3.73}

核心差异说明 #

• PySpark的meanSquaredError：直接以总样本数n为分母，公式为 SSE / n。示例中SSE=7654.61389，n=19，计算得 7654.61389 / 19 ≈ 402.8744，与模型输出一致。
• SAS Proc ANOVA的MSE及手动计算值：以**残差自由度（n - k - 1）**为分母，公式为 SSE / (n - k - 1)。示例中残差自由度=17，计算得 7654.61389 / 17 ≈ 450.2714，与SAS结果一致。

结论 #

这是两种工具对MSE的定义不同：PySpark的meanSquaredError是样本均方误差（除以n），而SAS的MSE是无偏估计的均方误差（除以残差自由度），并非PySpark的bug。若要在PySpark中得到与SAS一致的MSE，需手动用SSE除以残差自由度计算。

内容的提问来源于stack exchange，提问作者magooken