Validation accuracy for neural network

Question

When training a neural network, I usually plot the accuracy obtained on the validation data (validation accuracy) as an intermediate measure of the network's performance – the final measure being test accuracy. This validation accuracy is measured one or more times during a single epoch, depending on the size of the training data.

Usually, I stop training when the validation accuracy starts dropping – a sign of overfitting.

However, for large datasets I often don't have enough computational power to wait for the validation accuracy to drop. I merely stop the training after the validation accuracy (or validation loss) stays constant for a while, like so:

Technically, I do not know if the model can improve further, but is it reasonable to assume that it won't? Again, this is only for large datasets with limited computing power available. With sufficient computing power available, I can afford to wait until validation accuracy drops.

Answer 1

You can't assume the loss wouldn't drop with additional cycles, but it's generally safe to assume there are diminishing returns. This is why stopping rules are defined in terms of both number of iterations and change in the loss function. You want to stop when the gains become vanishingly small. At that point, increasing your accuracy from, say, 90.1% to 90.2% probably have little practical benefit, and performance on a hold-out sample probably won't differ much. But, that is an empirical question you can actually test. Run a model to near completion then run another with more stringent stopping criteria and compare on a hold-out.