机器学习优化算法分类咨询：不同分类体系下的类别归属疑问

Great question—these different classification frameworks aren't competing with each other; they're just ways to categorize optimization algorithms based on different core characteristics of either the algorithm itself, the problem it's solving, or the constraints applied. Let's break each down with clear examples so you can map common algorithms to each category:

1. Classification by Derivative Order (Algorithm-Centric) #

This categorization is based on what type of derivative information the algorithm uses to update parameters:

• First Order Optimization Algorithms: These only use the first derivative (gradient) of the loss function. They're computationally cheaper and scale well to large datasets/models.
  • Common examples: Stochastic Gradient Descent (SGD), Adam, Adagrad, RMSProp, Momentum SGD
• Second Order Optimization Algorithms: These leverage the second derivative (via the Hessian matrix, which describes curvature of the loss surface) to make more informed updates. They converge faster but are computationally expensive (especially for high-dimensional problems).
  • Common examples: Newton's Method, Quasi-Newton Methods (like BFGS, L-BFGS), Gauss-Newton Method

2. Classification by Objective Function Convexity (Problem-Centric) #

This is based on the shape of the loss/objective function you're optimizing, not the algorithm itself. A single algorithm can be used for both convex and non-convex problems:

• Convex Optimization: The objective function has no local minima other than the global minimum. These problems are easier to solve reliably.
  • Typical use cases: Linear regression, logistic regression (without regularization tricks), support vector machines (SVMs)
  • Algorithms used: SGD, Newton's Method, Proximal Gradient Descent
• Non-Convex Optimization: The objective function has multiple local minima/saddle points. Most modern ML problems fall into this category.
  • Typical use cases: Neural network training, deep learning models, matrix factorization (for recommendation systems)
  • Algorithms used: Adam, SGD with Momentum, RMSProp (first-order algorithms are preferred here due to computational efficiency)

3. Classification by Constraints (Problem-Centric) #

This categorization depends on whether there are explicit constraints on the parameters you're optimizing:

• Unconstrained Optimization: No limits on parameter values—you're free to update parameters in any direction to minimize the loss.
  • Example: Standard neural network training (without weight clipping or regularization as hard constraints)
  • Algorithms used: SGD, Adam, Newton's Method
• Constrained Optimization: Parameters must satisfy specific conditions (e.g., weights ≥ 0, L1/L2 regularization bounds, or equality constraints like sum-to-one for probabilities).
  • Example: SVMs with margin constraints, linear programming, training models with weight regularization (treated as a constraint)
  • Algorithms used: Proximal Gradient Descent, Projected SGD, Interior-Point Methods, Lagrange Multiplier-based methods

Key Note: These Classifications Overlap! #

It's important to remember that these are not mutually exclusive categories. For example:

• SGD is a first-order algorithm that can be used for convex (linear regression) or non-convex (neural networks) problems, and can be adapted for constrained optimization (via Projected SGD).
• Newton's Method is a second-order algorithm that works great for convex problems (like logistic regression) but is rarely used for non-convex problems (due to Hessian computation cost and risk of converging to local minima).

内容的提问来源于stack exchange，提问作者conjuring