5
$\begingroup$

I am currently training a CNN model by using cifar10 images (50000 for training, another 10000 for validation).

I plot training loss, validation loss and accuracy against training iteration: enter image description here

I am not sure when should I stop the training, should I stop it at around iteration 1250, since overfitting happens beyond that point? Or should I stop it at around iteration 5000, since I can get the maximum validation accuracy?

Why can the overfitted model have a higher validation accuracy?

$\endgroup$
  • $\begingroup$ How large is your test set? $\endgroup$ – Peter Jun 16 at 19:10
  • $\begingroup$ @Peter 50000 for training, another 10000 for validation, this is only the training, therefore no test set was used. $\endgroup$ – Lei Xun Jun 18 at 20:58
  • $\begingroup$ maybe a look at a test set, which has not been used for training or evaluation, would help to get a better idea if the large difference in scores is really sustainable. If this works, I see no reason why one should have doubt about performance. $\endgroup$ – Peter Jun 18 at 21:01
  • $\begingroup$ @Peter Do you mean using a test set to see which model is actually better? Can I use the validation set as the test set? Since the model did not learn from it. $\endgroup$ – Lei Xun Jun 18 at 21:07
1
$\begingroup$

I will try to answer your question "Why can the overfitted model have a higher validation accuracy?" with an example, then try to answer the remaining questions (TLDR at bottom):

Say we have 2 classes with a 1:1 ratio in both the training and testing sets (each 4 instances).

Perhaps our training/validation curves are just like yours (I highlighted some points for reference). training/validation curve example

Going from point A to point B ($\overline{AB}$), we do not observe overfitting:

  • validation accuracy increases
  • validation loss decreases

$\therefore$ We might conclude an inverse relationship between validation accuracy/loss.

However, $\overline{BC}$ shows overfitting:

  • validation accuracy is nearly constant
  • validation loss increases

This appears to contradict the relationship we observed with the $\overline{AB}$ example!

The reason behind this phenomenon is simple: we cannot assume a linear dependence between the accuracy and loss functions.


Let us define our accuracy and loss functions respectively:

$accuracy:\quad 1-\frac{1}{N}\sum^N_{i=1}|[X_i]-\hat{X}_i|$

$loss \space (MSE):\quad \frac{1}{N}\sum^N_{i=1}(X_i-\hat{X}_i)^2$

$\{X \space | \space 0<X<1\},\qquad\{\hat{X} \space | \space \{0,1\}\}$

We can see that these are linearly independent (which you could verify by calculating their Wronskian). How might this affect the relationship between training/loss curves?

Let us plug-in some numbers to see. Columns {A,B,C} represent the model output vectors for these three epochs and the corresponding ground truth is in $\hat{X}$ column.

\begin{array}{|c|c|c|c|} \hline A & B & C & \hat{X}\\ \hline 0.6 & 0.3 & 0.3 & 0\\ \hline 0.2 & 0.2 & 0.2 & 0 \\ \hline 0.4 & 0.4 & 0.1 & 1 \\ \hline 0.8 & 0.8 & 0.8 & 1 \\ \hline \end{array}

\begin{array}{|c|c|c|} \hline epoch & accuracy \space calculation & accuracy & loss \space calculation & loss \\ \hline A & 1-\frac{1}{4} * (1 + 0 + 1 + 0) & 0.50 & \frac{1}{4} * (0.6^2 + 0.2^2 + 0.6^2 + 0.2^2) & 0.2000\\ \hline B & 1-\frac{1}{4} * (0 + 0 + 1 + 0) & 0.75 & \frac{1}{4} * (0.3^2 + 0.2^2 + 0.6^2 + 0.2^2) & 0.1325 \\ \hline C & 1-\frac{1}{4} * (0 + 0 + 1 + 0) & 0.75 & \frac{1}{4} * (0.3^2 + 0.2^2 + 0.9^2 + 0.2^2) & 0.2450 \\ \hline \end{array}

Therefore, $C_{loss} > A_{loss},\quad$even though$\quad C_{accuracy} > A_{accuracy}$!

Of course this is a very simple example, and your situation will have many more complexities, but establishing the linear independence of the validation accuracy and validation loss and understanding these curves is a critical aspect of determining which epoch model to select.


TLDR: With this in mind, you should use the model instance at epoch B. This is the point at which overfitting starts, not at epoch 1250. (although I suggest saving many intermediate models and empirically determining their viability on another dataset).

After epoch B, your validation accuracy is not changing while your training loss decreases. Therefore, your weights are fitting more to the training data without improving your results, ergo overfitting. At epoch 1250, your model is still improving validation accuracy quite dramatically, so stopping training then would yield a very poor model.

As we can see from your validation accuracy curve, this model will probably perform similarly to models beyond epoch B, but there are a couple reasons you might prefer model B.

If you ever wanted to resume training (perhaps you add more data to your training set but do not want to restart completely), then having weights that are not overly fitted (or even vanished!) to the initial dataset would be ideal.

A very similar reason is that, even though this particular dataset shows virtually no difference between the epoch B model and models beyond epoch B, other datasets might. It would then be easier to adjust model B for these new datasets rather than an overfitted model.

$\endgroup$
  • $\begingroup$ Thanks for the reply. I wonder if we could use validation loss to find the over-fitting point? Previous I thought the gap between training loss and validation loss is the over-fitting point. $\endgroup$ – Lei Xun Jun 24 at 17:33
  • $\begingroup$ You start overfitting when your results on the validation set either worsen or remain constant while your training loss decreases. The "results" on the validation set could mean loss but it could also mean accuracy. Generally loss is more granular, so it is a better indicator. With this definition, we can see that after epoch B, neither validation accuracy nor validation loss improves, so that is where the model starts overfitting. It is therefore not the gap that determines overfitting, rather the derivatives of the curves. $\endgroup$ – Benji Albert Jun 24 at 18:17
  • $\begingroup$ I see now. Thanks. $\endgroup$ – Lei Xun Jun 24 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.