0
$\begingroup$

When the validation error of my Neural Network that I am trying to train is slowly decreasing but not by much, is it okay to stop train the network at that point, or do I need to increase the training time until the minimum validation error is reached?

For instance, in the last 5 epochs my validations errors are shown below:

| end of epoch   1 | time: 3782.50s | valid loss 6.7914 | valid ppl 890.1194
| end of epoch   2 | time: 3802.14s | valid loss 6.6084 | valid ppl 741.2616
| end of epoch   3 | time: 3791.33s | valid loss 6.5249 | valid ppl 681.8797
| end of epoch   4 | time: 3792.55s | valid loss 6.4513 | valid ppl 633.5318
| end of epoch   5 | time: 3804.15s | valid loss 6.3884 | valid ppl 594.8927

so like between the 4th epoch and the 5th epoch, the loss decreased by ~0.975% (= (6.4513-6.3884)/6.4513 * 100)? would it be okay to stop training the network at this point?

Thank you,

$\endgroup$

3 Answers 3

0
$\begingroup$

You should keep training.

In many scenarios ~1% decrease in validation loss a bid deal in itself. However, looking at the trend it looks like you validation loss is set to decrease by more than ~1%, if you let it train form, say, 20 more epochs. The decreases will get smaller and smaller, but they will accumulate.

Generally, you should continue training if your validation loss is decreasing.

$\endgroup$
3
  • $\begingroup$ Hello, thank you for your reply. what do you exactly mean when you say "in many scenarios ~1% decrease in validation loss is a bid deal in itself"? can you elaborate more on that? thanks, $\endgroup$
    – HDB
    Mar 13, 2020 at 1:07
  • $\begingroup$ Sure, validation loss is very closely associated (well hopefully!) with the performance of the model, and decreasing 1% loss can be a lot of money in a real world application, or it can be big boost to a score in a kaggle competition. $\endgroup$
    – Akavall
    Mar 13, 2020 at 1:14
  • $\begingroup$ Thank you, your answer is helpful $\endgroup$
    – HDB
    Mar 13, 2020 at 1:17
0
$\begingroup$

You can keep the epoch high but add early stopping in your code. Early stoping will stop the training when validation loss stops decreasing significantly.

$\endgroup$
0
$\begingroup$

In the paper Early Stopping___ But When? they perform a series of experiments that explore the relationship between final model performance and the number of epochs to wait before stopping early.

The TL;DR is that when you use a small num_epochs you are more likely to be settling for a local minimum.

As a rule of thumb, you don't want to stop training if your loss is continuing to decrease every epoch.

So yeah, keep training!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.