1
$\begingroup$

I am reading about conjugated gradient methods to understand how they exactly work. I understand that a pair of vector $u$ and $v$ are conjugated with respect to $A$ if $u^TAv=0$. I also read that $A$ is symmetric, positive definite matrix.

I am trying to find out how is that related to training of neural network by minimalising mean-square error function using CG method. What will be $A$ matrix in that case? How $A$ matrix is connected to weights of neural network. And is it still symmetric and positive definite? I read What is conjugate gradient descent? this thread and resources linked there, but I still can't figure it out.

I'm sure I'm missing something simple, but could you give me a bit of explanation?

Thank you,

Max

$\endgroup$
2
$\begingroup$

The traditional conjugate gradient descent is an increment on the gradient descent that just takes a direction that is fully orthogonal to the previous descent direction. There is no $A$ matrix in that case.

There are different rules (you can check some in my old optimization toolbox at https://github.com/mbrucher/scikit-optimization/blob/master/scikits/optimization/step/conjugate_gradient_step.py). If I remember properly FR combined with strong Wolfe-Powell line search rule give one of the best answer. The issue is that it requires more computation, which is why line search is never used in neural networks optimization.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I see. Every new change direction is fully orthogonal to gradient vector. It is realised by line search (in many implementations, I like Polak-Ribiere's one). But how is orthogonality of vectors defined without that matrix? $\endgroup$ – maksylon Jan 2 '19 at 20:37
  • $\begingroup$ Simple scalar product. That's the basic definition (equivalent to $A=I$). $\endgroup$ – Matthieu Brucher Jan 2 '19 at 20:52
  • $\begingroup$ Ah. And it is the exact thing my brain lacked. Thank you! Matthieu!. Good luck with your work. I wish I had enough reputation to up you. $\endgroup$ – maksylon Jan 2 '19 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.