I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was wondering why more advanced methods don't seemed to be used, such as SQP algorithms or interior-point methods. Is it because training a neutral net is always a simple unconstrained optimization problem, and the above-mentioned methods would be unnecessary? Any insight would be great, thanks.

  • 2
    $\begingroup$ Because the more expensive methods don't offer enough advantage over simple gradient descent. Or maybe we do not know how to harness them well enough. Why gradient descent works as well as it does is still debated; cf. e.g. The Marginal Value of Adaptive Gradient Methods in Machine Learning. Welcome to the site! $\endgroup$
    – Emre
    Feb 22 '18 at 17:24
  • $\begingroup$ @Emre Thanks for your answer. Don't you think GD approaches using momentum perform so much better? $\endgroup$ Feb 22 '18 at 18:12
  • 1
    $\begingroup$ It has for me; momentum functions as a dampener enabling the optimizer to power through rough patches of the loss surface, but here we have a paper that questions this folk wisdom. I'll keep using it until the dust settles. $\endgroup$
    – Emre
    Feb 22 '18 at 18:15
  • $\begingroup$ Excuse me sir, @Emre If you want to train a network from scratch based on what you have referred to, you would prefer GD over Adam? $\endgroup$ Feb 23 '18 at 13:24
  • $\begingroup$ I would not, because GD needs tuning, and Adam will beat untuned GD. When I hear "advanced methods" I think of (quasi) second order or natural gradients. $\endgroup$
    – Emre
    Feb 23 '18 at 17:29

In my reply here

Does gradient descent always converge to an optimum?

it is explained that standard gradient descent works well because backtracking gradient descent works well (proven in our recent paper mentioned in the post) and in the long run backtracking gradient descent behaves like the standard gradient descent.

The main issue with other methods, I think, is that they require too strong conditions for convergence or no convergence is proven at all. In both cases, these make them less applicable to realistic applications.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.