1
$\begingroup$
  1. When filepath="weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5" I get key error val_acc (I use tensorflow 1.14.0).
  2. When filepath="weights.best.hdf5" and save_best_only=True, it checkpoints the very best model observed during training, however it failed to save the model because accuracy was not increasing, so does it mean I need to increase the epochs. Also why doesn't it consider the available accuracy scores and pick the maximum score as the best model and save.

Sample Code

# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, init= "uniform" , activation= "relu" ))
model.add(Dense(8, init= "uniform" , activation= "relu" ))
model.add(Dense(1, init= "uniform" , activation= "sigmoid" ))
# Compile model
model.compile(loss= "binary_crossentropy" , optimizer= "adam" , metrics=[ "accuracy" ])
# checkpoint
filepath="weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor= "val_acc" , verbose=1, save_best_only=True,
mode= "max" )
callbacks_list = [checkpoint]
# Fit the model
model.fit(X, Y, validation_split=0.33, nb_epoch=50, batch_size=10,
callbacks=callbacks_list, verbose=2)

Train on 514 samples, validate on 254 samples
Epoch 1/50
 - 0s - loss: 0.6846 - accuracy: 0.6401 - val_loss: 0.6691 - val_accuracy: 0.6732
Epoch 2/50
 - 0s - loss: 0.6672 - accuracy: 0.6401 - val_loss: 0.6517 - val_accuracy: 0.6732
Epoch 3/50
 - 0s - loss: 0.6600 - accuracy: 0.6498 - val_loss: 0.6503 - val_accuracy: 0.6772
Epoch 4/50
 - 0s - loss: 0.6529 - accuracy: 0.6440 - val_loss: 0.6401 - val_accuracy: 0.6811
Epoch 5/50
 - 0s - loss: 0.6476 - accuracy: 0.6654 - val_loss: 0.6346 - val_accuracy: 0.6772
Epoch 6/50
 - 0s - loss: 0.6385 - accuracy: 0.6459 - val_loss: 0.6448 - val_accuracy: 0.6299
Epoch 7/50
 - 0s - loss: 0.6334 - accuracy: 0.6615 - val_loss: 0.6242 - val_accuracy: 0.6772
Epoch 8/50
 - 0s - loss: 0.6261 - accuracy: 0.6498 - val_loss: 0.6166 - val_accuracy: 0.6693
Epoch 9/50
 - 0s - loss: 0.6216 - accuracy: 0.6673 - val_loss: 0.6057 - val_accuracy: 0.6969
Epoch 10/50
 - 0s - loss: 0.6192 - accuracy: 0.6673 - val_loss: 0.6059 - val_accuracy: 0.6654
Epoch 11/50
 - 0s - loss: 0.6247 - accuracy: 0.6595 - val_loss: 0.5972 - val_accuracy: 0.6811
Epoch 12/50
 - 0s - loss: 0.6139 - accuracy: 0.6518 - val_loss: 0.5936 - val_accuracy: 0.6811
Epoch 13/50
 - 0s - loss: 0.6107 - accuracy: 0.6732 - val_loss: 0.5908 - val_accuracy: 0.6772
Epoch 14/50
 - 0s - loss: 0.6093 - accuracy: 0.6770 - val_loss: 0.5848 - val_accuracy: 0.6929
Epoch 15/50
 - 0s - loss: 0.6001 - accuracy: 0.6829 - val_loss: 0.5866 - val_accuracy: 0.6772
Epoch 16/50
 - 0s - loss: 0.6022 - accuracy: 0.6829 - val_loss: 0.5804 - val_accuracy: 0.7008
Epoch 17/50
 - 0s - loss: 0.5957 - accuracy: 0.6790 - val_loss: 0.5990 - val_accuracy: 0.6811
Epoch 18/50
 - 0s - loss: 0.5911 - accuracy: 0.6887 - val_loss: 0.6046 - val_accuracy: 0.6575
Epoch 19/50
 - 0s - loss: 0.6028 - accuracy: 0.6770 - val_loss: 0.5706 - val_accuracy: 0.7008
Epoch 20/50
 - 0s - loss: 0.6086 - accuracy: 0.6673 - val_loss: 0.5790 - val_accuracy: 0.7205
Epoch 21/50
 - 0s - loss: 0.5904 - accuracy: 0.6965 - val_loss: 0.5636 - val_accuracy: 0.7008
Epoch 22/50
 - 0s - loss: 0.5931 - accuracy: 0.6965 - val_loss: 0.6001 - val_accuracy: 0.6654
Epoch 23/50
 - 0s - loss: 0.5895 - accuracy: 0.7023 - val_loss: 0.5647 - val_accuracy: 0.7087
Epoch 24/50
 - 0s - loss: 0.5837 - accuracy: 0.7101 - val_loss: 0.5628 - val_accuracy: 0.7283
Epoch 25/50
 - 0s - loss: 0.5837 - accuracy: 0.6965 - val_loss: 0.5584 - val_accuracy: 0.7047
Epoch 26/50
 - 0s - loss: 0.5828 - accuracy: 0.6887 - val_loss: 0.5593 - val_accuracy: 0.7362
Epoch 27/50
 - 0s - loss: 0.5913 - accuracy: 0.6965 - val_loss: 0.5580 - val_accuracy: 0.6890
Epoch 28/50
 - 0s - loss: 0.5861 - accuracy: 0.6965 - val_loss: 0.5597 - val_accuracy: 0.7441
Epoch 29/50
 - 0s - loss: 0.5846 - accuracy: 0.6887 - val_loss: 0.5584 - val_accuracy: 0.6850
Epoch 30/50
 - 0s - loss: 0.5780 - accuracy: 0.6946 - val_loss: 0.5553 - val_accuracy: 0.7165
Epoch 31/50
 - 0s - loss: 0.5802 - accuracy: 0.6887 - val_loss: 0.5619 - val_accuracy: 0.7323
Epoch 32/50
 - 0s - loss: 0.5816 - accuracy: 0.6965 - val_loss: 0.5574 - val_accuracy: 0.6929
Epoch 33/50
 - 0s - loss: 0.5740 - accuracy: 0.7140 - val_loss: 0.5540 - val_accuracy: 0.6850
Epoch 34/50
 - 0s - loss: 0.5723 - accuracy: 0.7004 - val_loss: 0.5523 - val_accuracy: 0.7323
Epoch 35/50
 - 0s - loss: 0.5746 - accuracy: 0.6965 - val_loss: 0.5645 - val_accuracy: 0.7323
Epoch 36/50
 - 0s - loss: 0.5684 - accuracy: 0.7004 - val_loss: 0.5664 - val_accuracy: 0.7323
Epoch 37/50
 - 0s - loss: 0.5726 - accuracy: 0.7140 - val_loss: 0.5492 - val_accuracy: 0.7087
Epoch 38/50
 - 0s - loss: 0.5628 - accuracy: 0.7218 - val_loss: 0.5933 - val_accuracy: 0.6850
Epoch 39/50
 - 0s - loss: 0.5832 - accuracy: 0.6965 - val_loss: 0.5487 - val_accuracy: 0.7087
Epoch 40/50
 - 0s - loss: 0.5604 - accuracy: 0.7140 - val_loss: 0.5780 - val_accuracy: 0.7087
Epoch 41/50
 - 0s - loss: 0.5714 - accuracy: 0.7082 - val_loss: 0.5590 - val_accuracy: 0.6929
Epoch 42/50
 - 0s - loss: 0.5745 - accuracy: 0.7004 - val_loss: 0.5550 - val_accuracy: 0.6929
Epoch 43/50
 - 0s - loss: 0.5623 - accuracy: 0.7140 - val_loss: 0.5497 - val_accuracy: 0.7047
Epoch 44/50
 - 0s - loss: 0.5697 - accuracy: 0.7062 - val_loss: 0.5497 - val_accuracy: 0.7205
Epoch 45/50
 - 0s - loss: 0.5618 - accuracy: 0.7140 - val_loss: 0.5485 - val_accuracy: 0.7205
Epoch 46/50
 - 0s - loss: 0.5614 - accuracy: 0.7121 - val_loss: 0.5457 - val_accuracy: 0.7126
Epoch 47/50
 - 0s - loss: 0.5587 - accuracy: 0.7140 - val_loss: 0.5548 - val_accuracy: 0.7205
Epoch 48/50
 - 0s - loss: 0.5584 - accuracy: 0.6984 - val_loss: 0.5489 - val_accuracy: 0.7323
Epoch 49/50
 - 0s - loss: 0.5619 - accuracy: 0.6984 - val_loss: 0.5561 - val_accuracy: 0.6969
Epoch 50/50
 - 0s - loss: 0.5813 - accuracy: 0.7043 - val_loss: 0.5551 - val_accuracy: 0.7165
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Can you elaborate on Q2 in particular what makes you think it didn't pick the best 'score'? Can you share some output/evidence? $\endgroup$
    – fswings
    May 15, 2021 at 18:53
  • $\begingroup$ I have added the output to my question, I used 50 epochs, at epoch 28, the val_accuracy has improved to 0.7441, but the model was not checkpointed. Even after finishing 50 epochs the model was not saved. But when I use save_best_only = False, then the model is checkpointed at every epochs. $\endgroup$
    – Mathew
    May 16, 2021 at 20:02

2 Answers 2

Reset to default
1
$\begingroup$

For Question 1, replace val_acc with val_accuracy since the metric is named as accuracy. This might also solve your 2nd question.

...

filepath="weights-improvement-{epoch:02d}-{val_accuracy:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor= "val_accuracy" , verbose=1, save_best_only=True,
mode= "max" )

...
Improve this answer
$\endgroup$
0
$\begingroup$

For Question 1, it appears you may be using f-string but haven't specified the f before the quotation mark:

filepath=f"weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5

For Question 2, based on your output, I agree with @spb below, that you need to change want you monitor to monitor='val_accuracy' if this is the metric you want to use.

If you need to improve your model, there are plenty of existing answers that tackle approaches to try and improve an existing model.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.