2
$\begingroup$

The choice of optimiser and how data is scaled are both very important things in machine learning, yet they are not hyperparameters (as far as I am aware). It is also not necessarily obvious which scaling is best.

Should we trial a simple or same version of the model we wish to train over all optimisers and data scalings to figure which one is best, then use this scaling and optimiser to perform training, hyperparameter optimisation and final testing? i.e. would the 'best optimiser and scaling' generally hold across choice of hyperparameters? And if not, are there formal ways to go about these choices (beyond no scaling yielding data of different magnitudes and scaling roughly sorting out that problem), or should we just use a general intuition/strategy of choice based on our specific problem, more as above?

$\endgroup$
1
  • $\begingroup$ optimizers have ton of hyperparameters... for the scaling, probably depends on the use case and on the model that you are using $\endgroup$
    – anon
    Jun 26, 2022 at 13:38

1 Answer 1

1
$\begingroup$

If you are working on a project that is similar to plenty of other ones, then you should apply what usually works the best, because such a project has been redone over and over again with different algorithms, and people eventually found out which one works the best.

However, if you are working on an innovation project with no similarity to another one on the Internet, you should first understand optimization algorithms and data scaling techniques as best as possible, because even if you run plenty of models in parallel to know which one is the best, there is no guarantee that it would be the case in the long term.

In addition to that, algorithms are also related to the data scaling: a stochastic gradient descent will not behave the same as an Adam optimizer with the same data, because they don't differentiate the data in the same way.

There could be even better algorithms like AdamW but it depends on the data.

Some algorithms may also be more sensitive to noise, some others to high variability.

In the end, if you know an algorithm very well, you could also transform your data accordingly to reach extraordinary results, for instance by removing noise with smoothing techniques or applying a log to deal with high variability.

There is a paper that describes quite well different optimizers that is worth reading: enter image description here

Source: https://www.semanticscholar.org/paper/A-Survey-of-Optimization-Methods-From-a-Machine-Sun-Cao/3119ea9c7ad7a5e044dc7c267329a4bbf00d0158

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.