1
$\begingroup$

What role do 'Solvers' play in optimization problems? Surprisingly, I could not find any definition for 'Solvers' online. All the sources I've referred to just explain the types of solvers & the conditions under which each one is supposed to be used.

Examples of Solvers - ['Newton-cg', 'lbfgs', 'liblinear', 'sag,' 'saga']

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The title refers to Machine Learning but the body of your question does not. Can you clarify if and how your question relates to ML? $\endgroup$
    – Sammy
    May 21 at 14:33
  • $\begingroup$ Solvers are used in ML algorithms like Logistic Regression, SVM, etc. $\endgroup$
    – Apoorva
    May 21 at 15:51

2 Answers 2

Reset to default
1
+100
$\begingroup$

In other words, you would like to have a clear comparison between all the ML optimisation algorithms you've mentionned?

Here is a quite extended list of diffetent ML optimisation methods, including the ones you've mentionned:

enter image description here

enter image description here

Source: https://www.semanticscholar.org/paper/A-Survey-of-Optimization-Methods-From-a-Machine-Sun-Cao/3119ea9c7ad7a5e044dc7c267329a4bbf00d0158

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much:). This is what I've been looking for for a very long time, a crisp & clear description of optimization methods. $\endgroup$
    – Apoorva
    May 23 at 10:32
2
$\begingroup$

Disclaimer: I'm not especially knowledgeable about this topic, this answer presents my modest understanding.

As far as I know, there is no formal definition for solver simple because this a broad term which describes any system meant to solve a set of constraints or equations. Since every solver must have its own specific scope of problems that it can solve, very different systems can potentially be called "solvers". But the term is more common in the context of complex optimization problems, for example in the domain of multi-objective optimization. Also Prolog is the most famous example of declarative programming: the program is expressed as constraints that the Prolog engines solves.

To my knowledge the term "solver" is rarely used in the context of Machine Learning, probably because:

  • ML algorithms don't always rely on optimization, in the sense that many algorithms are completely deterministic.
  • Even the ML methods which rely on optimization are not "complex" in the sense that they only deal with very specific types of problems with numerical constraints, i.e. they are not general solvers which can take various types of constraints into account.

Some ML methods like genetic algorithms are specifically used for optimization problems, but (I think) that they are not called solvers because they rely on randomization, not mathematical methods.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the response. By 'Solvers' I mean - ['Newton-cg', 'lbfgs', 'liblinear', 'sag,' 'saga']. $\endgroup$
    – Apoorva
    May 21 at 18:03
  • 1
    $\begingroup$ @Apoorva if your question is specific to these methods, I don't understand what kind of answer you're looking for? It's actually normal that you don't find a definition of 'solver' which corresponds exactly to these methods, the concept is much more general than this. $\endgroup$
    – Erwan
    May 22 at 11:03
  • $\begingroup$ I simply want to know the purpose of these Solvers -['Newton-cg', 'lbfgs', 'liblinear', 'sag,' 'saga'] that are popularly used in the 'Hyper-parameter Tuning' of ML algorithms. I'm not looking for a detailed explanation of each of these Solvers, just a one or few lines of basic description that tells what these Solvers exactly do. For example, Gradient Descent can be defined as an optimization algorithm that minimizes a loss function by getting optimal parameters. Similarly, I'm looking for a simple definition of the Solvers listed above. Hope I'm clear now. $\endgroup$
    – Apoorva
    May 22 at 13:05
  • 1
    $\begingroup$ @Apoorva I'm not knowledgeable enough to answer, but I think you should probably ask a specific question about each of these models, if only because it's unlikely that somebody would be expert enough to know all of them. And actually I think asking on stats.stackexchange.com could be a good idea, because this is a more theoretical and advanced question. $\endgroup$
    – Erwan
    May 23 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.