3
$\begingroup$

I read the article on Vanishing Gradient problem, which states that the problem can be rectified by using ReLu based activation function.

Now I am not able to understand that if using ReLu based activation function solves the problem, then why there are so many research papers suggesting the use of Newton's method based optimization algorithms for deep learning instead of Gradient Descent? While reading research papers, I was having the strong impression that vanishing gradient problem was the core reason for such suggestions but now I am confused whether Newton's method is really needed if Gradient Descent can be modified to rectify all the problems faced during machine learning.

$\endgroup$
2
$\begingroup$

Why there are so many research papers suggesting the use of Newton's method based optimization algorithms for deep learning instead of Gradient Descent?

Newton method has a faster convergence rate than gradient descent, and this is the main reason why it may be suggested as a replacement for gradient descent.

Is Newton's method really needed if Gradient Descent can be modified to rectify all the problems faced during machine learning?

Existence of vanishing gradient problem depends on the choice of "activation function" and the "depth" of network. Newton method and gradient descent would both face this problem for a function like Sigmoid, since in the flat extremes of Sigmoid both first and second order derivatives are small and exponentially vanishing by depth. In other words, the problem is solved for both methods by the choice of function.

As a side note, 1st- and 2nd-order derivatives of Sigmoid go to zero at the same rate. Here is a graph of Sigmoid and its derivatives; zoom into the extremes.

Historical note. Newton method predates the vanishing gradient problem (which was faced after the introduction of Backpropagation in 60s) by centuries.

| improve this answer | |
$\endgroup$
  • $\begingroup$ In this paper, section 2.1 says that scale invariance is important as it eliminates the need to tweak the learning rate, then why scalability of gradient descent is preferred over the scale invariance of newton method. $\endgroup$ – Aman Mar 20 '19 at 15:35
  • $\begingroup$ @Aman removed the controversial parts $\endgroup$ – Esmailian Mar 20 '19 at 15:51
  • $\begingroup$ which is more worse based on efficiency , vanishing gradient in newton method or vanishing gradient on gradient descent. I'm new to machine learning, so I would appreciate if you can clear or atleast guide me with the above doubt. thank you. $\endgroup$ – Aman Mar 20 '19 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.