I am reviewing some course material where the lecturer suggests that instead of guessing the learning rate parameter in gradient descent implementation, one could use the inverse of the Hessian multiplied by the negative of the Jacobian, to determine the step-size.

Any help with the intuition behind using the inverse of the Hessian would be much appreciated.


It's not an intuition, it's mathematics. And more precisely, the quadratic expansion of a function.

Say you have $f(x)=f(x_n+\Delta x)=f(x_n)+f'(x_n)\Delta x + f''(x_n)\Delta x^2$ and you want to get to a point where $f'(x) = 0$, then you will use $\Delta x=-\frac{f'(x_n)}{f''(x_n)}$.

Generalizing to n dimension, $f''$ is the Hessian of the equation.

So of course this works perfectly for least squares, it will only "work" close to the solution for non quadratic functions.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.