1
$\begingroup$

I'm a little bit confused how loss function is calculated in neural network training. There's is said that in theory when using Grid Search or Monte Carlo methods we can calculate all the possible loss function values. But obviously this requires too many resources and is not a good way for neural network training. Alternatively, when using Gradient Descent we have a possibility to evaluate a single value to know in which direction we should go in order to test for the next value. We then could climb down the ladder one step by another until we reach the optimal value.

enter image description here

But on the other hand, in the following PyTorch example there is said that the loss function is calculated on the basis of all predicted and real values. And after that the gradient is calculated. So, what's the point of Gradient Descent in this way when all the loss values in previous step are calculated anyway?

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Gradient descent is based on sources: your data and your loss function.

In supervised learning, at each training step the predictions of the Network are compared with the atcual, true results. The value of a loss function is calcualated, which is telling your model "how wrong it is". At this point, the weights of the Network must be updated accordingly.

In order to do that, a formula based on the chain rule of derivatives calculates retrospectively the contribution of each weight to the final loss value. The value of each weight is then changed, based on their impact on the loss function (mathematically, it's a first partial derivative for each weight). This process is called backpropagation, since it logically starts from the bottom of the Network and is computed backwards up to the input layer.

This process has to be done for each of the Network's learnable weights. The higher the number of parameters, the higher the number of partial derivatives that are computed at each training iteration for the weight update. When all the weights are updated, the position of the Gradient will (hopefully) be a small step closer to the global minimum of the loss function.

Coming to your doubts: grid search on the loss function values is something you can do only in theory, not in practice. It could take an impossible amount of time to do that, even for the most powerful computers. At the moment, the Gradient descent algorithm is the main tool to train Neural Networks. As far as I know, other methods such as Monte Carlo and Genetic algorithms are feasible but not practical, therefore not state-of-the-art.

I suggest you two great posts to understand how Gradient descent and backprop work: this one by Andrej Karpathy and this one by Colah.

Improve this answer
$\endgroup$
6
  • $\begingroup$ So what's the difference between Grid Search and Gradient Descent if in both cases all the loss values are calculated for each individual data point? If we already know the loss value for each individual data point, then why not just take the minimum of them? $\endgroup$
    – Tauno
    Commented Sep 23, 2019 at 11:52
  • $\begingroup$ Grid search and Gradient descent are very different things. Gradient descent is an optimization technique that starts from a random point of the objective function, and moves iteratively on it trying to find the global minimum. Grid search on the other side is a much more rough and simple technique: it means "try n possible values on the parameter space and see what's better than others". It's a very inefficient, time consuming, brute force approach that in practice doesn't make training possible. (You can use grid search for hyperparameter tuning, but that's a completely different thing.) $\endgroup$
    – Leevo
    Commented Sep 23, 2019 at 12:27
  • $\begingroup$ I could understand the difference if the Gradient Descent takes only single random loss value as a starting point and then iteratively finds optimum path. But as I understand before calculating gradients the complete hyperparameter space for all the loss values are calculated. And therefore I can't see where the effect is achieved compared to Grid Search. They both calculate loss values for each individual data point. Confusing. $\endgroup$
    – Tauno
    Commented Sep 23, 2019 at 13:14
  • $\begingroup$ "before calculating gradients the complete hyperparameter space for all the loss values are calculated." Sorry this information is not correct. It's impossible to explore the either the full hyperparameter and the parameter spaces. Is it clear for you the difference between hyperparameters and parameters? $\endgroup$
    – Leevo
    Commented Sep 23, 2019 at 13:16
  • $\begingroup$ Ok, maybe I didn't express myself correctly. What's my point is that if in Gradient Descent the loss values are calculated for each individual data point then it could also take enormous amount of resources. Haven't still got an answer in which way it differs from Grid Search in this respect (both calculate all possible loss values). $\endgroup$
    – Tauno
    Commented Sep 23, 2019 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.