A situation that validation accuracy is higher than test accuracy at its minimum loss region

Question

This case would be an example.

Around epoch = 800, we have the validation loss(orange line) reaches its minimum region.

• Should we record the accuracy of the model as an average value from epoch 780 to 800? Or shall we take the accuracy after it became an almost straight line (after epoch = 1200)?

• If we record the accuracy at the minium point of validation loss. The accuracy may not be the highest. For example, at epoch = 800, val_loss = 0.627,val_accuracy = 0.783. At epoch=909, val_loss = 0.624, val_accuracy = 0.761

what is the best way to do it for such a situation?

Thanks

Answer 1

Assuming the goal is to give the true performance of the model, then it should be the performance obtained by applying the final model on the test set. The fact that the performance on the validation set can be observed during training is irrelevant.

So the question becomes: which model should you select based on observing the performance on the validation set during training? You can select the model which gives the maximum accuracy, but the real performance is obtained when you evaluate it on a fresh test set.

The second question depends on which performance measure you want to optimize: if you want to optimize accuracy, then you should pick the model which obtains the highest accuracy.