9
$\begingroup$

I'm trying to train a model in keras and I'm using ModelCheckpoint to save the best model according to a monitored validation metric (in my case the Jaccard index).

While I can see the model improving in tensorboard, when I try to load the weights and evaluate the model it isn't working at all. Furthermore, by the timestamp on the file where the weights are supposed to be stored, I can tell that they are not being saved at all. The timestamp corresponds roughly to the time I started training.

Has anyone encountered such a problem before?

$\endgroup$
8
$\begingroup$

Do you run ModelCheckpoint on its default parameters (besides monitor)?

ModelCheckpoint has a parameter called mode which specifies the type of metric to be used. mode can take 3 values 'min' 'max' and 'auto' (which is the default):

  • min: means that you want to minimize the metric (e.g. the loss function).
  • max: means you want to maximize the metric (e.g. accuracy).
  • auto: attempts to figure what to do on its own. If you look at the code, it checks if the metric's name contains 'acc' or if it starts with 'fmeasure'. If yes it sets the mode to max, if not it sets it to min.

In your case, you monitor the jaccard index, which is a metric you want maximized, so you want the mode set to max. Normally because "jaccard" contains the string "acc", even if the mode is set to auto it should work fine.

If however you named your metric something arbitrary (e.g. my_metric), the default mode will be set to min, which means that it will store the weights that achieve the least performance on your metric, which should be the weights of the first epoch.

Suggestion: next time try with mode='max' to be sure.

$\endgroup$
2
  • $\begingroup$ Yes, you are right. I had named my metric intersection_over_union and it was probably storing weights that had the lowest score, which were the ones from the first epoch. That makes a lot of sense. Thanks! $\endgroup$ – ILM91 Sep 16 '18 at 22:24
  • $\begingroup$ You're welcome. Glad I could help! $\endgroup$ – MzdR Sep 16 '18 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.