1
$\begingroup$

Say you come across a loss curve as shown below. At which loss should you trust the model? The initial lucky guess or after it has stabilized?

And more importantly, why?

$\endgroup$
2
  • $\begingroup$ with lucky guess you mean the peak after the dashed line? $\endgroup$
    – Leevo
    Jun 26, 2019 at 13:08
  • $\begingroup$ I mean the peak at the dashed line, where the loss is the lowest. $\endgroup$ Jun 26, 2019 at 13:09

2 Answers 2

2
$\begingroup$

The loss in the curve suggests that the training can be improved by tuning hyperparameters, especially the learning rate and/or the batch size. Therefore the optimal decision would be to keep refining the model instead of using the already trained model.

If tuning hyperparameters is not an option, at least you should re-split training and validation data or at very least repeat with a different random seed.

If none of those is an option, you should take the best validation loss. The reason is that we are assuming that there is no leak of the validation data to the training data and that the validation data is representative of the data the model is going to be tested with. Without any other evidence, we should assume that the "sweet spot" you found would also lead to better results in a piece of test data the model has not seen before.

$\endgroup$
1
$\begingroup$

You should always look at validation loss, you don't care about training when evaluating your model's performance.

But your idea is quite in line with the principle of early stopping is: keep training the model, checking its performance at each epoch; once you find a best loss value, save the model; stop training once you don't find any loss improvement for a number of epochs defined by your patience hyperparameter.

However, about your specific problem I agree with @ncasas, i.e. that looking at your image it seems your model can be improved.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.