Conjugated gradient method. What is an A-matrix in case of neural networks

I am reading about conjugated gradient methods to understand how they exactly work. I understand that a pair of vector $$u$$ and $$v$$ are conjugated with respect to $$A$$ if $$u^TAv=0$$. I also read that $$A$$ is symmetric, positive definite matrix.

I am trying to find out how is that related to training of neural network by minimalising mean-square error function using CG method. What will be $$A$$ matrix in that case? How $$A$$ matrix is connected to weights of neural network. And is it still symmetric and positive definite? I read What is conjugate gradient descent? this thread and resources linked there, but I still can't figure it out.

I'm sure I'm missing something simple, but could you give me a bit of explanation?

Thank you,

Max

1 Answer

The traditional conjugate gradient descent is an increment on the gradient descent that just takes a direction that is fully orthogonal to the previous descent direction. There is no $$A$$ matrix in that case.

There are different rules (you can check some in my old optimization toolbox at https://github.com/mbrucher/scikit-optimization/blob/master/scikits/optimization/step/conjugate_gradient_step.py). If I remember properly FR combined with strong Wolfe-Powell line search rule give one of the best answer. The issue is that it requires more computation, which is why line search is never used in neural networks optimization.

• I see. Every new change direction is fully orthogonal to gradient vector. It is realised by line search (in many implementations, I like Polak-Ribiere's one). But how is orthogonality of vectors defined without that matrix? Jan 2 '19 at 20:37
• Simple scalar product. That's the basic definition (equivalent to $A=I$). Jan 2 '19 at 20:52
• Ah. And it is the exact thing my brain lacked. Thank you! Matthieu!. Good luck with your work. I wish I had enough reputation to up you. Jan 2 '19 at 20:57