0
$\begingroup$

I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function. What is really bothering me that for a non-convex function like $y = -x^2$, how gradient descent is actually calculated as here descent would go to minus infinity instead of directly converging to global maxima. Thus, it contradicts the point of getting stuck in a saddle point for a function like $(x^2 - y^2)$.

$\endgroup$
0
$\begingroup$

The calculation of the gradient is the same no matter the optimization algorithm you use. Gradient descent is a method to find (local) minima of a function. If no such minimum exists, any algorithm to find it will render wrong results or never converge. $-x^2$ only has a maximum, so looking for a minimum does not make much sense and to find the maximum you would need to run gradient ascent, i.e. going uphill.

Getting stuck in a saddle point is a different issue. For $f(x,y) = x^2 - y^2$ the gradient is $\nabla f(x,y) = 2[x, -y]^T$ and thus equals zero at $x=y=0$, so gradient ascent/descent would just sit there and not move. But $f(0,0) = 0$ is neither a minimum nor a maximum ($f$ actually doesn't have either of these).

| improve this answer | |
$\endgroup$
  • $\begingroup$ if I use ∇f(x,y) (where f(x,y)=x2$−y2$ is my loss function) to calculate gradient descent with respect to each dimension, which is x, y in this case, I will reach local minima with respect to x, but what about y? Do we calculate gradient ascent wrt y? But that doesn't make any sense. $\endgroup$ – Mayank Mittal Oct 27 '19 at 14:07
  • 1
    $\begingroup$ For a function with multi-dimensional arguments talking about "local minima with respect to x" does not make much sense in the first place, unless you consider y=const. And if you end up at x=y=0 for f(x,y) = x^2 - y^2 depends on your starting position, since gradient descent updates both x and y in each step. As I wrote in the answer, f(x,y) = x^2 - y^2 does not have maxima or minima, so running gradient descent or ascent does not make sense. $\endgroup$ – matthiaw91 Oct 27 '19 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.