It is said that second order optimization methods in neural networks work better than first order because they contain information about rate of change of gradient or the curvature. This information helps to choose a better step size for moving forward on the error surface. It is not clear as to how the rate of change of gradient is controlling the step size and leading to better optimization. For simplicity, consider only one weight update iteration.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
You need to consider two steps of your first order optimisation process to see why a second order method can be usefull. (For more clarity we'll work in one dimension).
First step : calculate the derivative and move your evaluation point accordingly. Then, second step : calculate the derivative and move your evaluation point accordingly. If the derivative in the second step is bigger / lower than in the first step, you'll move less/more in the second step than in the first step. So the rate of change of the derivative along your path impact the change of your step size. So if you had information on the rate of change of the derivative in the frist step it may have been better to incorporate it directly in your first step calculation. In the end it is rather intuitive that the second derivative may able to find better suited step sizes.
In practice it is a bit more complex as the actual gain in performance depends on how the second order derivative is calculated / approximated. (See here)
answered Jan 5, 2020 at 1:06
-
$\begingroup$ Need a small clarification, if the derivative in the second step is more, it means there is a saddle point, so we need to move less and vice versa? $\endgroup$ Jan 5, 2020 at 18:38
-
$\begingroup$ You seems right, I used to work on maximisation problems, so may wording may not have followed my mind. $\endgroup$ Jan 5, 2020 at 19:05