I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was wondering why more advanced methods don't seemed to be used, such as SQP algorithms or interior-point methods. Is it because training a neutral net is always a simple unconstrained optimization problem, and the above-mentioned methods would be unnecessary? Any insight would be great, thanks.
$\begingroup$
$\endgroup$
6
-
2$\begingroup$ Because the more expensive methods don't offer enough advantage over simple gradient descent. Or maybe we do not know how to harness them well enough. Why gradient descent works as well as it does is still debated; cf. e.g. The Marginal Value of Adaptive Gradient Methods in Machine Learning. Welcome to the site! $\endgroup$– EmreCommented Feb 22, 2018 at 17:24
-
$\begingroup$ @Emre Thanks for your answer. Don't you think GD approaches using momentum perform so much better? $\endgroup$– Green FalconCommented Feb 22, 2018 at 18:12
-
1$\begingroup$ It has for me; momentum functions as a dampener enabling the optimizer to power through rough patches of the loss surface, but here we have a paper that questions this folk wisdom. I'll keep using it until the dust settles. $\endgroup$– EmreCommented Feb 22, 2018 at 18:15
-
$\begingroup$ Excuse me sir, @Emre If you want to train a network from scratch based on what you have referred to, you would prefer GD over Adam? $\endgroup$– Green FalconCommented Feb 23, 2018 at 13:24
-
$\begingroup$ I would not, because GD needs tuning, and Adam will beat untuned GD. When I hear "advanced methods" I think of (quasi) second order or natural gradients. $\endgroup$– EmreCommented Feb 23, 2018 at 17:29
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
In my reply here
Does gradient descent always converge to an optimum?
it is explained that standard gradient descent works well because backtracking gradient descent works well (proven in our recent paper mentioned in the post) and in the long run backtracking gradient descent behaves like the standard gradient descent.
The main issue with other methods, I think, is that they require too strong conditions for convergence or no convergence is proven at all. In both cases, these make them less applicable to realistic applications.
