1
$\begingroup$

I am currently writing a scientific thesis which consists of two parts. In the first part I am building ML models with neural networks, support vectors etc. and the second part is about finding global minima for optimization objectives with optimization methods like Particle Swarm Optimization, Ant Colony Optimization, Simulated Annealing etc.

When writing about the parts in the introductory part, I would write machine learning methods when referring to the first part and metaheuristic optimization methods when referring to the second part.

Is it possible to distinguish them like this?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ almost all machine learning algorithms use some kind of optimisation method (eg gradient descent, back-propagation etc). Optimisation is more general than its application in machine learning $\endgroup$
    – Nikos M.
    Mar 7, 2021 at 19:37
  • $\begingroup$ It would probably be more precise to classify methods by type of task: supervised vs. unsupervised, classification vs. regression... $\endgroup$
    – Erwan
    Mar 8, 2021 at 0:01
  • $\begingroup$ @NikosM. That's the reason why I do not know how to distinguish them correctly, because both use some kind of optimisation method. Could I say that one optimization aims to map an objective function and the other aims to minimize a self-defined objective function? $\endgroup$
    – Emma
    Mar 8, 2021 at 11:37

1 Answer 1

Reset to default
3
+50
$\begingroup$

In a wider context every machine learning method can be re-cast as some type of optimisation problem. For example for Neural Networks the associated optimisation problem is "find the weights which minimise some loss function of the data given an architecture". This is solved using back-propagation (which is a layered gradient-descent method) and when minima of the loss function are found we say the systen has "learned".

On the other hand there is nothing stopping us from re-casting optimisation problems as "learning" problems

. A possible differentiation (based on what you mention in the question) is between analytic methods and non-analytic methods.

A. Analytic Methods: Gradient-based, Primal-Dual etc... (eg NNs, SVMs)

B. Non-Analytic Methods: Particle systems, Genetic/Evolutionary systems, Simulation methods, Stochastic methods, etc..

The basic differrence is both whether the form of the objective function is known and if analytic methods (eg gradient-descent vs particle systems) are used to find optima.

Again, there is nothing, in principle, stopping us from using these non-analytic optimisation methods to do machine learning.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your detailed answer, it helps me a lot. Another question about the ML methods: as you said, their optimization problem is to map an unknown objective function. However, the ML methods have another objective function, in regression often the Mean Squared Error between the output values of the real system and the ML model. When writing this I am afraid that the reader can get confused - how would you distinguish them so that it's clear? $\endgroup$
    – Emma
    Mar 13, 2021 at 9:00
  • $\begingroup$ Per this answer there is no fundamental difference between classification and regression. Regression again uses some objevtive function (like MSE) but other than that the optimisation strategy falls under these categories. Unless I miss something in your comment so you can explain in more detail $\endgroup$
    – Nikos M.
    Mar 13, 2021 at 16:24
  • $\begingroup$ Sorry, the question was not clearly formulated. In my thesis I want to avoid that the function you call "loss function" (e.g. MSE) is confused with the unknown objective function which is to be mapped by the ML model. Did you call it extra "loss function" and not objective function to avoid this confusion? $\endgroup$
    – Emma
    Mar 14, 2021 at 10:05
  • 1
    $\begingroup$ Loss function is another name for objective function. In the sense that the minima of the loss function correspond to the model learning. Loss function is an objective function that measures for example accuracy loss or prediction error. The input-output mapping IS NOT an objective function that is optimised, in fact the input-output mapping is learned when the objective function that represents some accuracy measure (eg MSE) is minimised. Hope these clarifications are helpful $\endgroup$
    – Nikos M.
    Mar 14, 2021 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.