I have a few basic questions about tracking losses during training.

  1. If I am using mini-batch training, should I validate after each batch update or after I have seen the entire dataset?
  2. What should be the condition to stop the training to prevent overfitting? Do you save the model at that point?
  3. In case I use mini-batch training losses fluctuate a lot, depending on the random choice of training data, and sometimes validation loss is less than training loss. Is this normal? I think my confusion about this point will be answered in 1 by itself.

1 Answer 1

  1. Both approaches can be done: I recommend to validate after every batch when your are just playing around with your gradient method, to see if the validation accuracy goes up or down and to figure out how everything is going.

  2. In this setting, you can adopt early stopping, although there are many ways to prevent from overfitting. Early stopping checks, at the end of an epoch, your validation accuracy, and saves the model if it is the best so far. If the validation accuracy does not increase in the next $n$ epochs (and here $n$ is a parameter that you can decide), then you keep the last model you saved and stop your gradient method.

  3. Validation loss can be lower than training loss, this happens sometimes. In this case, you can state that you are not overfitting.

  • $\begingroup$ Hey thanks!! it clears a lot of confusion for me, however, I actually meant to say that my validation loss is lower than training loss. I will make that edit $\endgroup$
    – pg2455
    Commented Apr 26, 2018 at 21:17
  • $\begingroup$ Validation loss can be lower than training loss, this has happened to me many times, specially in deep learning when using dropout. $\endgroup$ Commented Apr 26, 2018 at 21:18
  • $\begingroup$ So how do you decide about overfitting? $\endgroup$
    – pg2455
    Commented Apr 26, 2018 at 21:47
  • $\begingroup$ If your validation loss is lower than your training, then you can be sure that you are not overfitting $\endgroup$ Commented Apr 27, 2018 at 5:21
  • 1
    $\begingroup$ Does the minibatch-Validation step imply performing Cross Validation too? $\endgroup$
    – Kari
    Commented Apr 27, 2018 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.