5
$\begingroup$

What is Conjugate Gradient Descent of Neural Network? How is it different from Gradient Descent technique?

I came across a resource, but was unable to understand the difference between the two methods. It has mentioned in the procedure that:

the next search direction is determined so that it is conjugate to previous search directions.

What does this mean? Also, what is line search mentioned in the web page?

Can anyone please explain it with the help of a diagram?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ This paper, while long, builds up the CGD from simple linear algebra. $\endgroup$
    – Jérémie Clos
    Commented Sep 30, 2015 at 13:11
  • $\begingroup$ The wikipedia article actually does a pretty good job of illustrating the difference between conjugate gradient method and the gradient descent method and even approaches conjugate gradient from the perspective of both a direct solve and an iterative solve. If you don't like math, then Andrew Ng does a pretty good job of describing both with essentially no math in his Coursera Machine Learning Course $\endgroup$
    – AN6U5
    Commented Sep 30, 2015 at 16:36
  • $\begingroup$ If I have a curve, like a bell say, then how will one apply conjugate gradient? I went through the wiki, did not understand. I wanted a pictorial description $\endgroup$
    – girl101
    Commented Oct 1, 2015 at 3:37

1 Answer 1

Reset to default
5
$\begingroup$

What does this sentence mean?

It means that the next vector should be perpendicular to all the previous ones with respect to a matrix. It's like how the natural basis vectors are perpendicular to each other, with the added twist of a matrix:

$\mathrm {x^T A y} = 0$ instead of $\mathrm{x^T y} = 0$

And what is line search mentioned in the webpage?

Line search is an optimization method that involves guessing how far along a given direction (i.e., along a line) one should move to best reach the local minimum.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.