硕士论文无文献回归问题：机器学习最优模型判定咨询

Hey there! Great job testing out all those regression models for your thesis—especially since there’s no existing literature to guide you here. Figuring out the "optimal" model when performance is so tied to hyperparameter tuning is a super common (and tricky) problem, but there’s a structured way to tackle it. Here’s my step-by-step advice:

Before you start comparing models, make sure your evaluation process is fair and reliable:

• Pick consistent, meaningful metrics: For regression, don’t just rely on one metric. Use a combination like RMSE (penalizes large errors), MAE (robust to outliers), and R² (explains variance). Align these with your problem’s real-world impact—if outliers matter more (e.g., predicting rare high-value events), prioritize MAE over RMSE.
• Use proper cross-validation: Ditch the single train-test split, especially if your dataset is small. For most regression tasks, 5- or 10-fold cross-validation works well. If you’re working with time-series data (since you’re using LSTMs), use time-series split instead of random folds—you don’t want to train on future data to predict the past!
• Guard against data leakage: This is critical. Always apply preprocessing steps (like scaling, normalization) within each cross-validation fold, not on the entire dataset upfront. Leaking test-set information will make your results look better than they actually are.

Don’t waste time tweaking parameters by hand—automate the process to ensure fairness across all models:

• Use automated tuning tools: For small parameter spaces, GridSearchCV works, but for larger spaces (like LSTM layer sizes or Random Forest n_estimators), RandomizedSearchCV is more efficient. For even better results, try Bayesian optimization libraries like Optuna or scikit-optimize—they learn from previous trials to focus on promising parameter combinations.
• Assign equal computational resources: Give every model the same amount of time or number of tuning iterations. If you spend 3 days tuning your LSTM but only 1 hour on SVR, your comparison will be biased. Fairness here is key.

Looking at a single performance number isn’t enough—you need to account for variability and significance:

• Aggregate cross-validation results: Report the mean performance across folds plus the standard deviation. For example, Model A might have an RMSE of 5.0 ± 0.2, while Model B has 4.8 ± 0.8. Even though B’s mean is better, A is far more stable—critical for real-world generalization.
• Test for statistical significance: Use tests like the paired t-test or Wilcoxon signed-rank test to check if the performance difference between two top models is actually meaningful, not just random chance. If the difference isn’t statistically significant, lean toward the simpler model (e.g., SVR over LSTM) for interpretability and ease of use.

Your optimal model isn’t just about raw performance—it should fit your research goals:

• Prioritize interpretability: Master’s theses often value explainability. Models like Random Forest (with feature importance scores), Decision Trees, or even SVR (via support vectors) are easier to explain than black-box models like LSTMs or ANNs. If your research requires justifying why predictions are made, a slightly less performant but interpretable model might be better.
• Consider computational cost: If you plan to do follow-up experiments (like ablation studies or scaling to larger datasets), a lightweight model (e.g., SVR, KNN) will be easier to work with than resource-heavy LSTMs.
• Lean into novelty: If a less commonly used model (like LSTM for your specific regression task) performs comparably to others, highlighting it can add novelty to your thesis—just make sure you can justify why it’s a good fit for your problem.

Once you’ve narrowed down to 2-3 top models, test them on a holdout dataset that you’ve kept completely separate (no training, no tuning, no preprocessing applied to it until this step). This is your final check to ensure the model generalizes to unseen data. The model that performs best here is your optimal choice.

内容的提问来源于stack exchange，提问作者MerklT