openMDAO中SLSQP优化算法运行机制及迭代过程咨询

Let’s break down exactly what’s happening with SLSQP in your OpenMDAO setup, using your paraboloid example to clarify the behavior you’re seeing:

1. Why You See Finite Difference Gradient Calculations #

SLSQP requires first-order gradients to generate search directions, but it does not compute explicit second-order derivatives (which is why you don’t observe them). Here’s the breakdown:

• Since you didn’t provide an analytical gradient for your paraboloid, OpenMDAO automatically uses forward finite differences to approximate the gradient. The "iterations" you see (x=0.01,y=0 and x=0,y=0.01) aren’t actual optimization iterations—they’re function evaluations to calculate df/dx and df/dy at the current point (x=0,y=0).
• SLSQP triggers gradient calculations at the start of each main optimization iteration, or if the line search process needs to refine the direction.

2. The Initial Jump to (5.99, -8.01) #

That first big jump is a steepest descent step (negative gradient direction):

• From your finite difference results, you calculated df/dx=5.99 and df/dy=-8.01. For minimization, the optimal initial search direction is the negative gradient, which points directly to (0 + 5.99, 0 + (-8.01))—exactly the next point you observed.
• SLSQP uses a line search to find the optimal step length along this direction. In your case, the step landed you very close to the paraboloid’s minimum, hence the large drop in f to -25.9597.

3. The Next Iteration’s Jump to (8.372726, -6.66007) #

Once near the minimum, SLSQP switches from steepest descent to using a quasi-Newton approximation of the Hessian matrix (typically BFGS updates) to build a more accurate search direction:

• After recalculating the gradient at (5.99, -8.01), SLSQP updates its approximate Hessian using the difference between the current and previous gradients/positions.
• It then solves a quadratic programming (QP) subproblem: minimize a quadratic approximation of your objective function (no constraints in your case). The solution to this QP gives the next search direction.
• The final point (8.37..., -6.66...) comes from taking a line search step along this QP-derived direction. If this point seems to drift away from the minimum, it’s likely due to:
  • A poor Hessian approximation early in the iterations (common when starting far from the minimum)
  • Line search parameters allowing an overly large step
  • Numerical noise from finite difference gradients

4. Occasional "Big Jumps Without Derivative Calculations" #

These are almost always due to line search failures or Hessian instability:

• If the current Hessian approximation doesn’t reflect the true curvature of your function, the QP subproblem may generate a direction that moves you away from the minimum.
• If the line search can’t find a step that satisfies the Wolfe conditions (standard criteria for acceptable steps), SLSQP may fall back to a larger default step or retry with a revised gradient.

Key Resources to Deepen Your Understanding #

• Nocedal & Wright's Numerical Optimization: The definitive textbook on SLSQP and quasi-Newton methods—Chapter 18 covers sequential quadratic programming in detail.
• Scipy's SLSQP Documentation: Explains the implementation specifics (like default line search parameters and Hessian update rules) used by OpenMDAO.
• OpenMDAO Optimization Guide: Details how OpenMDAO wraps Scipy's solvers, including gradient handling and iteration logging.

Tips to Debug and Visualize SLSQP's Logic #

• Distinguish main iterations from gradient evaluations: Enable OpenMDAO's detailed iteration logging (show_iter=True in your optimizer setup) to see which steps are actual optimization iterations vs. gradient function calls.
• Use analytical gradients: Implementing an explicit gradient for your paraboloid will eliminate finite difference noise and make it easier to track how SLSQP uses gradient information. For your paraboloid, the gradient is straightforward (e.g., if f = ax² + bxy + cy² + dx + ey + f, df/dx = 2ax + by + d, df/dy = bx + 2cy + e).
• Print gradient and Hessian data: Use OpenMDAO's debugging tools to output the gradient and approximate Hessian at each iteration—this will let you directly see how the search direction is calculated (search direction = inverse(Hessian) × (-gradient)).

内容的提问来源于stack exchange，提问作者Jeroen