I often use "early stopping" when I train neural nets, e.g. in Keras:
from keras.callbacks import EarlyStopping # Define early stopping as callback early_stopping = EarlyStopping(monitor='loss', patience=5, mode='auto', restore_best_weights=True) # ...THE MODEL HERE... # Call early stopping in .fit history = model.fit_generator(..., callbacks=[early_stopping])
Question: I often wonder if it is better to monitor the loss (
monitor='loss') or the validation loss (
monitor='val_loss'). Are there some takeaways from the literature? What is best practice?
My intuition would be that monitoring the validation loss gives a more direct feedback of the learning process, since in case the validation loss does not increase any further after some epochs, there seems to be little more to learn.
On the other hand - when I use
dropout for instance - the loss will often "lag behind" the validation loss (
val_loss). In this case, I believe that there still is something to learn for the model, even if this does not immediately translate into improvements in the validation loss.