General Q&A for Newcomers

[andrewrosemberg]

Following a great suggestion from @amontoison, I’m opening this Discussion/Q&A thread for newcomers to ask general questions about theory and implementation. I hope it becomes a helpful space—especially for those of us (myself included!) who are still getting familiar with the ecosystem.

[andrewrosemberg]

I’m trying to confirm that I understand the motivation behind having a dedicated AbstractNLSModel type in the NLPModels ecosystem.

My current understanding is that the main reason for distinguishing AbstractNLSModel from a general AbstractNLPModel is that specialized least-squares solvers (e.g. CaNNOLeS) can directly query residual-level information—such as through functions like NLPModels.jprod_residual!—instead of reconstructing it from the full objective.

Is this correct?

And as a follow-up: is the intention that an AbstractNLSModel should always expose the same API as a regular AbstractNLPModel, so that a general NLP solver can still solve it if needed? Or are general NLS models expected to be handled only by solvers that understand the residual API?

[amontoison]

Yes @andrewrosemberg, the main advantage of an AbstractNLSModel is that we can exploit the structure of the objective ||F(x)||^2 in the NLS solvers.

Examples:

• We can approximate the Hessian of the objective by J_F' * J_F (Gauss-Newton method) or J_F' * J_F + λI (Levenberg–Marquardt method). It works well in practice and it cheaper in terms of AD.

• We can convert the NLS into another one called FeasibilityFormNLS where we add the constraints F(x) - r = 0 and the objective is ||r||^2. If the problem is already constrained and / or the NLS solver supports constraints, this formulation is preferred.
  We can also use efficiently some NLP solvers with this reformulation.
  We will probably have more constraints than variables (LICQ doesn't hold) so it is not a good idea to call a vanilla NLP solvers on it (like Ipopt) but MadNCL will handle it without issue.

• To determine if an NLP problem is infeasible, we can solve an NLS problem with this reformulation and an NLS solver is expected to more efficient than a NLP solver on it.

I think I already partially answer you second question, but the key point is that depending on the solver that you want to use, you may want to only implement the NLS API or both NLS / NLP API for AbstractNLSModel.

With the recent release 0.21.6 of NLPModels.jl, we can specify in the NLPModelsMeta / NLSMeta what functions of the API is implemented or not, it will help a lot the solvers to decide what algorithms / options to use (see the related PR).
It will also help to determine if some solvers like MadNCL.jl can be used for NLS problems.

It is also relevant for:

• UnoSolver.jl when we don't have hprod;
• NLPModelsIpopt.jl / NLPModelsKNITRO.jl when we don't have the Hessian of the Lagrangian;
• MadNLP.jl when we don't have jtprod...

It is already deployed and exploited in UnoSolver.jl (solver), NLPModelsIpopt.jl (solver), and Vecchia.jl (model).
It is also relevant for NLPModelsJuMP.jl (model) and MadNLP.jl (solver) because the new MOI.VectorNonlinearOracle doesn't support operator-vector products.