如何解决线性回归模型中两个显著自变量的多重共线性问题？

Alright, let's work through this multicollinearity problem you're facing with your linear regression model targeting G3. You've got 32 predictors, and while G1 and G2 both come out as significant, their correlation of 0.85 is high enough to cause issues like unstable coefficient estimates and inflated standard errors. Here's how you can address this and land on your optimal model:

1. Combine Highly Correlated Variables #

Since G1 and G2 share so much variance, you can merge them into a single feature that retains their predictive power without the collinearity:

• Simple Aggregation: Create an average score like (G1 + G2)/2 or a weighted average if one has more domain relevance (e.g., weight G2 higher if it's a more recent assessment).
• Principal Component Analysis (PCA): Extract the first principal component from G1 and G2. This component captures the maximum shared variance between the two, and replacing G1/G2 with this single component eliminates collinearity entirely. Just make sure to interpret the component in your data's context (e.g., "overall prior performance").

2. Use Regularized Regression for Variable Selection & Collinearity Control #

With 32 predictors, regularized methods are perfect—they handle collinearity and trim redundant variables in one go:

• LASSO Regression (L1 Regularization): This method shrinks coefficients of less important variables to zero, effectively removing them from the model. It will automatically choose between G1 and G2 (or keep neither if they're redundant once other variables are considered). Use cross-validation to find the optimal regularization parameter alpha—in Python's scikit-learn, use LassoCV; in R, use glmnet with cross-validation.
• Ridge Regression (L2 Regularization): If you want to keep all variables but reduce collinearity's impact, Ridge shrinks coefficients towards zero without setting any to zero. It's great if both G1 and G2 have meaningful domain value and you don't want to drop either.

3. Domain-Driven Variable Removal #

Before jumping to automated methods, leverage your knowledge of the data:

• If G1 and G2 represent sequential assessments (e.g., G1 = midterm 1, G2 = midterm 2, G3 = final), G2 is likely a stronger predictor of G3 since it's more recent. Try dropping G1 and re-fitting the model—check if adjusted R² stays high, AIC/BIC decreases, and remaining coefficients stay stable.
• If they measure similar constructs (e.g., two different math scores), pick the one with higher correlation to G3 or better domain validity and remove the other.

4. Audit the Full Set of Predictors for Hidden Collinearity #

Don't stop at G1 and G2—32 variables mean there might be other collinear pairs. Calculate the Variance Inflation Factor (VIF) for each predictor:

• A VIF > 5 (or >10, depending on strictness) indicates significant collinearity. In Python, use statsmodels.stats.outliers_influence.variance_inflation_factor; in R, use the vif() function from the car package.
• Address all high-VIF pairs using the methods above (merge, remove, regularize) to ensure your final model is stable.

How to Pick the Optimal Model #

Evaluate candidate models using these metrics:

• Adjusted R²: Prioritize models with higher adjusted R² (it accounts for the number of predictors, unlike raw R²).
• AIC/BIC: Lower values mean a better balance of fit and model simplicity.
• Coefficient Stability: Check if coefficients change drastically when you add/remove predictors—stable coefficients mean less collinearity.
• Predictive Performance: Split your data into train/test sets and compare metrics like Mean Squared Error (MSE) or Mean Absolute Error (MAE) on the test set. The model with the best out-of-sample performance is often the most reliable.

内容的提问来源于stack exchange，提问作者WYQ