[RFC] API planning

[Vaibhavdixit02]

Broad Scope

This package will serve as the source for line search algorithms in SciML packages for nonlinear and optimization solvers. Multiple dispatch should be ensured so that downstream packages can use this package almost as a blackbox.

File/Code Organisation

1. An implementation of the Wolfe and Goldstein conditions - each as a struct. Associated dispatches for each for checking.
2. Each linesearch method will be defined as a struct with all the method specific hyperparameters and the common ones are arguments to the associated method dispatches
3. A kernel (function) for the method to be called in a loop, basically the update equation for the method.
4. Termination checking, no errors should be thrown but a nothing/NaN return when failure happens.
5. Extensions for supported AD packages.

[avik-pal]

We should also allow custom jac/jvp/vjp functions to be provided via kwargs and ignore them if they are not needed in the algorithm.

[Xiaoyi-Qu]

I notice that line search method is implemented in NonlinearSolve.jl. So, I think we can consider integrating this part into LineSearch.jl as well if possible.