I am learning Newton's method for second-order optimization in ML. I encountered this formula, but I do not understand how we get it. I guess it is from the Taylor series, but I still cannot fully explain this formula.
$$f(x + \Delta x) \approx f(x) + \langle \nabla f(x), \Delta x \rangle + { 1 \over 2} \langle \Delta x, B(x) \Delta x \rangle$$
$$B(x) = \nabla^2 f(x)$$
Yes, this is exactly the start of the multi-dimensional Taylor formula. You can reduce that to the scalar formula by considering the value evolution along a line $x+tv$, setting $\phi(t)=f(x+tv)$. Then the start of the scalar Taylor formula gives $$ \phi(t)=\phi(0)+\phi'(0)t+\frac12\phi''(0)t^2. $$ By the chain rule you then get $$ \phi'(0)=\sum_i\frac{\partial f(x)}{\partial x_i}v_i $$ and $$ \phi''(0)=\sum_{i,j}\frac{\partial^2 f(x)}{\partial x_i\partial x_j}v_iv_j $$ Now one can arrange the quadratic expression as a vector-matrix-vector product $v^TH_fv$ or as a multi-linear form $f''(x)[v,v]$. The first is compact for calculations that stop with the second-order term, the other can easily be extended for higher-order derivatives.
