My knowledge is that gradient actually gets to the global minimum and gradient descent try to take steps to the direction that he judges to be the lowest. I know that calculate the gradient from complex functions is non trivial problem, but let's say that we are a using neural net with a quadratic cost function, would be that hard to calculate the gradient and get actual global minimum?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
You might be mistaking the gradient itself with the mathematical approach to find the critical points of a differentiable function.
In the latter approach, you take the derivative of the function with respect to its parameters and find the values of the parameters that make it zero. These points in parameter space are critital points, that is, where the function has either a local maxima, a local minima or a saddle point. In order to know the actual type, you have to take the second derivative.
The problem with applying such an approach to neural networks is not finding the gradient of the loss function with respect to the parameters; currently we have automatic differentiation software that computes the gradients for you. The actual problem is solving the equation where such gradient equals to zero. We don't know how to do that but in the trivial cases. Furthermore, this only gives you local optima, not global ones.
A solution to that problem is using numerical optimization techniques, like the gradient descent family. They basically explore the parameter space in the descending direction of the gradient of the loss function, hoping to reach a minimum. This cannot be assured to converge for non-convex problems. Nevertheless, in practice it works quite well.
The reasons why gradient descent techniques work well in non-convex problems are an active line of research.
