I am having trouble wrapping my brain around validation loss. It's my understanding that loss is calculated at the end of the feed forward in a NeuralNet and is used in back propagation to update the weights. But I also see validation loss as a metric and don't understand fully what it is for other than observation, is validation loss used in updating weights? Or is it simply a measurement of how far off your observations were at the current epoch?


1 Answer 1


Validation loss is the same metric as training loss, but it is not used to update the weights. It is calculated in the same way - by running the network forward over inputs $\mathbf{x}_i$ and comparing the network outputs $\mathbf{\hat{y}}_i$ with the ground truth values $\mathbf{y}_i$ using a loss function e.g. $J = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}(\mathbf{\hat{y}}_i, \mathbf{y}_i)$ where $\mathcal{L}$ is the individual loss function based somehow on the difference between predicted value and target.

is validation loss used in updating weights?

No. In fact that would be counter to the purpose of it.

Or is it simply a measurement of how far off your observations were at the current epoch?

Yes. The point of using a data set that you did not train on, is to measure how well your model is generalising to unseen records. Very often when building a predictive model, that is the main goal, and is actually more important than fitting to your training data (the fitting to data part is necessary for learning, but is not the goal). This is the case whenever you are building a model to use to make decisions on its predictions against new, previously unseen and unlabelled, data.

It is not safe to use the training data in the same role to check for generalisation. It is possible for many model types to learn the training data perfectly, but be terrible at predicting from new values that are not from the training set. This is something you will want to avoid, is often caused by overfitting, and neural networks will often overfit. Using a validation set is a way to monitor and help control against overfitting.

Neural networks (and other model types) typically use a validation set on every epoch, because training too long can cause over-fitting, and models don't recover from that, they just get worse form that point on. So it can save a lot of wasted effort to monitor validation loss, and stop training when it has not improved for a long while, or starts to get worse.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.