Early stopping and final Loss or weights of models

Question

In a deep model, I used the Early stopping technique as below in Keras:

from keras.callbacks import EarlyStopping

early_stopping = [EarlyStopping(monitor='val_loss',
                          min_delta=0,
                          patience=2,
                          verbose=2, mode='auto')]

model.fit(train_x, train_y, batch_size=batch_size,
                epochs=epochs, verbose=1,
                callbacks=early_stopping,
                validation_data=(val_x, val_y))

model.fit(train_x, train_y, batch_size=batch_size, 
                epochs=epochs, verbose=2, 
                callbacks=early_stopping,
                validation_data=(val_x, val_y))

Now, when I run this code, in the output it prints the loss value for training and validation of each epoch.

I set the patience=2 in the early stopping. So, it continues the training process two times after when the validation loss increased instead of decreased.

Some things like this:

Epoch 1/10
- 198s - loss: 99.7160 - val_loss: 123.0397 
Epoch 2/10
- 204s - loss: 78.7000 - val_loss: 109.0344 
Epoch 3/10
- 208s - loss: 65.4412 - val_loss: 78.0097 
Epoch 4/10
- 268s - loss: 61.9812 - val_loss: 79.0312
Epoch 5/10
- 298s - loss: 59.1124 - val_loss: 79.3397 
Epoch 6/10
- 308s - loss: 57.2200 - val_loss: 218.0397 
Epoch 00007: early stopping

In the end, what will be the final weights of the model and the Loss values? The final time of training or two times before it?

If it considers the final epoch, so should it be better if I set the patience as little as possible to overcome the overfitting?

Thank you

Answer 1

EarlyStopping

The final weights will be saved, not the weights where your patience parameter is triggered.

Looking at the documentation for EarlyStopping, it seems not to be involved with saving weights at any point - it isn't mentioned.

EDIT

Upon further investigation (reading the source code), it seems you can indeed save the best, using the EarlyStopper callback

The class can be initialised with the aprameters restore_best_weights, as seen here. Then at the end of training, when your waiting period has overshot the patience parameter, the model's weights are returned to be the best weights (weights of the model at the time of loewst validation loss:

if self.restore_best_weights:
    if self.verbose > 0:
        print("Restoring model weights from the end of the best epoch")
    self.model.set_weights(self.best_weights)

It does this by tracking a chosen metric and comparing it to the recorded best value. By default this will be the validation loss. Check out the course code and it's description of the class there.

If these parameters don't work with your code, you will need to upgrade to the latest version (master branch from GitHub). This new parameter was only added 10 days ago!

ModelCheckpoint

If you want to use a callback to prevent overfitting, have a look at the ModelCheckpoint callback class. This has options to save the model weights at given times during training, and will allow you to keep the weights of the model at the end of the epoch specifically where the validation loss was at its minimum. This is selected using the the save_best_only parameter - use it like this:

from keras.callbacks import EarlyStopper, ModelCheckpoint

checkpointer = ModelCheckpoint(filepath, monitor='val_loss', verbose=0,
                               save_best_only=False, save_weights_only=False,
                               mode='auto', period=1)

# your early stopper as before
early_stopper = ...

All callbacks must be placed in a list:

my_callbacks = [checkpointer, early_stopper]

# pass to model along with other parameters as you did already
model.fit(..., callbacks=my_callbacks, ...)

If you don't use save_best_only, the default behaviour is to save the model at the end of every epoch.

You can set the filepath by using certain dynamic variables available during training, so that the filenames have some useful information:

E.g. filepath is weights.{epoch:02d}-{val_loss:.2f}.hdf5

the model checkpoints will be saved with the epoch number and the validation loss in the filename.

Have a look at the linked documentation for how to use the other parameters of ModelCheckpoint.