1
$\begingroup$

I am reading about conjugated gradient methods to understand how they exactly work. I understand that a pair of vector $u$ and $v$ are conjugated with respect to $A$ if $u^TAv=0$. I also read that $A$ is symmetric, positive definite matrix.

I am trying to find out how is that related to training of neural network by minimalising mean-square error function using CG method. What will be $A$ matrix in that case? How $A$ matrix is connected to weights of neural network. And is it still symmetric and positive definite? I read What is conjugate gradient descent? this thread and resources linked there, but I still can't figure it out.

I'm sure I'm missing something simple, but could you give me a bit of explanation?

Thank you,

Max

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The traditional conjugate gradient descent is an increment on the gradient descent that just takes a direction that is fully orthogonal to the previous descent direction. There is no $A$ matrix in that case.

There are different rules (you can check some in my old optimization toolbox at https://github.com/mbrucher/scikit-optimization/blob/master/scikits/optimization/step/conjugate_gradient_step.py). If I remember properly FR combined with strong Wolfe-Powell line search rule give one of the best answer. The issue is that it requires more computation, which is why line search is never used in neural networks optimization.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I see. Every new change direction is fully orthogonal to gradient vector. It is realised by line search (in many implementations, I like Polak-Ribiere's one). But how is orthogonality of vectors defined without that matrix? $\endgroup$
    – maksylon
    Commented Jan 2, 2019 at 20:37
  • $\begingroup$ Simple scalar product. That's the basic definition (equivalent to $A=I$). $\endgroup$
    – Matthieu Brucher
    Commented Jan 2, 2019 at 20:52
  • $\begingroup$ Ah. And it is the exact thing my brain lacked. Thank you! Matthieu!. Good luck with your work. I wish I had enough reputation to up you. $\endgroup$
    – maksylon
    Commented Jan 2, 2019 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.