I'm reading this paper "Deep learning via Hessian-free optimization" by J. Martens, I am having difficulty figure out the following statement:

In the standard Newton's method, $q_{\theta}(p)$ is optimized by computing the $N\times N$ matrix $B$ and then solving the system $Bp = −\nabla f(\theta)$.

(section 3 of the paper)

Is there any theorem, or statement anywhere regarding why the above system needs to be solved to optimize the local approximation? I came across another paper that has a reference to J. Martens and has used the same statement.


If you take a look at section 2, it says

The central idea motivating Newton’s method is that $f$ can be locally approximated around each $\theta$, up to 2nd-order, by the quadratic: $$ f(\theta + p) \approx q_\theta(p) \equiv f(\theta) + \nabla f(\theta)^Tp + \frac{1}{2} p^TBp \, \, (1) $$ where $B = H(\theta)$ is the Hessian matrix of $f$ at $\theta$. Finding a good search direction then reduces to minimizing this quadratic with respect to $p$.

To minimize, you need to take the derivative of (1) with respect to $p$ and set it to zero:

$$\Rightarrow \nabla f(\theta) + Bp = 0$$

which is equivalent to $Bp = -\nabla f(\theta)$.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.