1
$\begingroup$

Is there any possibility that cost function might end up in local minima rather than global minima while implementing the Neural network using Tensorflow ?

$\endgroup$
  • $\begingroup$ Yes, and it is not related to the tool; it can happen with pyTorch too. Rather, it depends on the network, regularization, and optimization algorithm. There is a growing body of literature on the loss surfaces of neural networks. One rule of thumb is to use a regularized, overparameterized model. Regularization makes it easy to fit and the overparameterization takes care of the quality of the local optima. For technical details please review the literature. $\endgroup$ – Emre Apr 26 '18 at 19:30
  • $\begingroup$ Thanks for clarifying the doubt @Emre, could you please provide me a useful link to refer the literature. $\endgroup$ – deepguy Apr 26 '18 at 20:00
  • $\begingroup$ Here's a recent work: What do neural loss surfaces look like? (lecture) $\endgroup$ – Emre Apr 26 '18 at 21:35
1
$\begingroup$

Most of the critical points in a neural network are not local minima, as it can be seen in this question. Although it is not impossible to fall into a local minimum, the probability of it happening is so low that in practice it does not happen, except from very special cases as a single-layer perceptron.

Being local minima so hard to find, this means that it is highly unlikely to come across the global minimum in your optimization method. All that we do in deep learning is decrease the loss function to find fairly good parameters, but finding local and global minima is extremely unlikely.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.