层次贝叶斯模型与OLS模型的对比统计量及方法咨询

Great question—when moving from frequentist OLS to hierarchical Bayesian (HB) models, the summary metrics shift a bit, but there are robust, meaningful ways to compare the two. Let’s cover the key metrics and how to compute them:

1. Fit Metrics (Alternatives to AIC/BIC) #

OLS uses AIC/BIC to balance fit and model complexity, but these rely on frequentist likelihoods. For HB models, use these Bayesian counterparts:

• WAIC (Widely Applicable Information Criterion): A fully Bayesian metric that estimates out-of-sample predictive accuracy while accounting for model complexity. Lower values mean better fit.
• LOO-CV (Leave-One-Out Cross-Validation): A more reliable alternative to WAIC that uses cross-validation to estimate predictive performance, avoiding some of WAIC’s biases with small samples.

Calculation Example (Using ArviZ, a standard Bayesian analysis library): #

import arviz as az

# Assume you have a fitted HB model trace (e.g., from PyMC3/Stan)
waic_results = az.waic(trace, model)
loo_results = az.loo(trace, model)

# Print summary
print("WAIC Summary:\n", waic_results)
print("\nLOO-CV Summary:\n", loo_results)

You can compare these values to OLS’s AIC/BIC—while not perfectly equivalent, lower WAIC/LOO values indicate a better-performing model than higher AIC/BIC values for OLS.

2. Predictive Performance Metrics #

HB models excel at quantifying predictive uncertainty, so direct predictive comparisons are highly informative:

• MAE (Mean Absolute Error) / RMSE (Root Mean Squared Error): Compute these for both models’ predictions vs. actual data. For HB, use the mean/median of the posterior predictive distribution as point predictions.
• Log Predictive Density (LPD): Measures how well the model predicts each observation. Sum the LPD values across all data points—higher totals mean better predictive fit.

Calculation Example: #

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error

# OLS predictions (from your existing code)
ols_preds = result.predict(d_df.ix[:, :-1])

# HB posterior predictive mean (example for PyMC3)
ppc = pm.sample_posterior_predictive(trace, model=model)
hb_preds = np.mean(ppc['y'], axis=0)  # 'y' is your target variable

# Compute metrics
ols_mae = mean_absolute_error(d_df.ix[:, -1], ols_preds)
hb_mae = mean_absolute_error(d_df.ix[:, -1], hb_preds)

ols_rmse = np.sqrt(mean_squared_error(d_df.ix[:, -1], ols_preds))
hb_rmse = np.sqrt(mean_squared_error(d_df.ix[:, -1], hb_preds))

print(f"OLS MAE: {ols_mae:.3f}, HB MAE: {hb_mae:.3f}")
print(f"OLS RMSE: {ols_rmse:.3f}, HB RMSE: {hb_rmse:.3f}")

3. Parameter Uncertainty Comparison #

OLS gives you coefficient standard errors and confidence intervals, but HB models provide full posterior distributions. Key comparisons here:

• Posterior Mean vs. OLS Coefficients: Check if HB’s posterior means align with OLS coefficients—HB often shrinks extreme coefficients toward the population mean (a key advantage for hierarchical data).
• Credible Intervals vs. Confidence Intervals: HB’s credible intervals (e.g., 95% HDI—Highest Density Interval) represent the range where parameters are likely to lie, whereas OLS’s confidence intervals are about repeated sampling. Compare interval widths to see which model quantifies uncertainty more realistically.

Example of HDI Calculation: #

# Get 95% HDI for HB coefficients
hdi = az.hdi(trace, hdi_prob=0.95)
print("HB Coefficient 95% HDIs:\n", hdi)

# Compare to OLS confidence intervals
print("\nOLS Coefficient 95% Confidence Intervals:\n", result.conf_int())

4. Model Checking Tools #

Beyond metrics, use diagnostic plots to validate both models:

• OLS: Residual plots (check for homoscedasticity, normality)
• HB: Posterior Predictive Checks (PPCs) to see if the model generates data that matches your observed data. For example, plot observed vs. predicted values, or compare the distribution of residuals from the model to actual residuals.

PPC Example: #

az.plot_ppc(az.from_pymc3(trace=trace, posterior_predictive=ppc))

This plot will show how well your HB model’s predicted data overlaps with the real data—if they match closely, the model is well-calibrated.

Note on F-Statistics #

Unlike OLS, HB models don’t use F-statistics (which rely on frequentist null hypothesis testing). Instead, use Bayes Factors if you want to compare evidence for nested models, but be aware they can be computationally intensive and sensitive to prior choices. WAIC/LOO are generally more practical for most use cases.

内容的提问来源于stack exchange，提问作者Casper Ritmeester