2
$\begingroup$

Error surface

Also, can someone please explain why the descent happens in a direction perpendicular to the contour lines?

enter image description here

$\endgroup$
1
  • $\begingroup$ Descent is performed along the gradient, perpendicular to the level sets because it has the steepest the rate of change, guaranteeing convergence to at least a local extremum if one exists. Otherwise you could just wander around the loss surface aimlessly. $\endgroup$
    – Emre
    Jun 19 '17 at 18:05
1
$\begingroup$

Hinton refers to a shape like this, where clearly the horizontal cross-sections are ellipses (or circles).

w1 and w2 are the base area and E the height

(In 3D space things can be perpendicular to each other in two ways) So no matter how steep the gradient at one point is, we need to travel perpendicular (in one way) to the contour lines in order to reach the minimum of the bowl. Think about it :D

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.