Would it be okay to stop training my neural network?

Question

When the validation error of my Neural Network that I am trying to train is slowly decreasing but not by much, is it okay to stop train the network at that point, or do I need to increase the training time until the minimum validation error is reached?

For instance, in the last 5 epochs my validations errors are shown below:

| end of epoch   1 | time: 3782.50s | valid loss 6.7914 | valid ppl 890.1194
| end of epoch   2 | time: 3802.14s | valid loss 6.6084 | valid ppl 741.2616
| end of epoch   3 | time: 3791.33s | valid loss 6.5249 | valid ppl 681.8797
| end of epoch   4 | time: 3792.55s | valid loss 6.4513 | valid ppl 633.5318
| end of epoch   5 | time: 3804.15s | valid loss 6.3884 | valid ppl 594.8927

so like between the 4th epoch and the 5th epoch, the loss decreased by ~0.975% (= (6.4513-6.3884)/6.4513 * 100)? would it be okay to stop training the network at this point?

Thank you,

Answer 1

You should keep training.

In many scenarios ~1% decrease in validation loss a bid deal in itself. However, looking at the trend it looks like you validation loss is set to decrease by more than ~1%, if you let it train form, say, 20 more epochs. The decreases will get smaller and smaller, but they will accumulate.

Generally, you should continue training if your validation loss is decreasing.

Answer 2

You can keep the epoch high but add early stopping in your code. Early stoping will stop the training when validation loss stops decreasing significantly.

Answer 3

In the paper Early Stopping___ But When? they perform a series of experiments that explore the relationship between final model performance and the number of epochs to wait before stopping early.

The TL;DR is that when you use a small num_epochs you are more likely to be settling for a local minimum.

As a rule of thumb, you don't want to stop training if your loss is continuing to decrease every epoch.

So yeah, keep training!