I'm a little bit confused how loss function is calculated in neural network training. There's is said that in theory when using Grid Search or Monte Carlo methods we can calculate all the possible loss function values. But obviously this requires too many resources and is not a good way for neural network training. Alternatively, when using Gradient Descent we have a possibility to evaluate a single value to know in which direction we should go in order to test for the next value. We then could climb down the ladder one step by another until we reach the optimal value.
But on the other hand, in the following PyTorch example there is said that the loss function is calculated on the basis of all predicted and real values. And after that the gradient is calculated. So, what's the point of Gradient Descent in this way when all the loss values in previous step are calculated anyway?