3
$\begingroup$

The point of EarlyStopping is to stop training at a point where validation loss (or some other metric) does not improve.

If I have set EarlyStopping(patience=10, restore_best_weights=False), Keras will return the model trained for 10 extra epochs after val_loss reached a minimum. Why would I ever want this? Has this model not just trained for 10 unnecessary epochs? Wouldn't it make more sense to give me back the model that was trained at the lowest validation loss i.e. with restore_best_weights=True?

Would love to hear situations where doing those extra 10 epochs of training is better than not doing them.

$\endgroup$
2
  • 1
    $\begingroup$ Try here github.com/keras-team/keras/issues/11371 $\endgroup$
    – WBM
    Apr 12, 2021 at 13:29
  • 1
    $\begingroup$ Great thanks! It looks like this is the default behavior to save memory. Though, there seem to be no arguments for restore_best_weights=False other than that. $\endgroup$
    – codeananda
    Apr 13, 2021 at 10:25

1 Answer 1

2
$\begingroup$

The default value is restore_best_weights=False.

There may be two reasons for such a default value:

  • Memory and speed. This is discussed in this Keras issue. This is the key paragraph of the discussion:

    If you want to restore the weights that are giving the best performance, you have to keep tracks on them and thus have to store them. This can be costly as you have to keep another entire model in memory and can make fitting the model slower as well.

  • Backward compatibility. Initially, there was no such flag and the only implemented behavior was not to restore the best weights. When adding the flag, the sensible decision would be to give it a default value that makes the old code keep its behaviour instead of silently changing it. The commit where the flag was introduced is this one. This is only speculation, as I am not the author.

Despite the default value, I would say that, if you have no problem in keeping another copy of the model in memory (i.e. because it is a huge model), the most sensible value is restore_best_weights=True.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.