My training accuray is 1.0 but the predictions on the training data are wrong

Question

My neural network is not working right, and I am trying to find out what is up.

I inserted just three images to a transfer learning (mobilenet) neural network. The three images' classes are: array([[0., 0., 0., 1.], [0., 1., 0., 0.], [0., 1., 0., 0.]])

I did 50 epochs on these pictures and by the 20th epoch or so, the training accuracy stayed at 1.0:

Epoch 50/50 3/3 [==============================] - 6s 2s/step - loss: 1.3671 - acc: 1.0000 - val_loss: 1.3770 - val_acc: 0.0000e+00

Then when I went to predict the outcome of the same three images like so: predictions_test_2 = model_mn.predict(X, batch_size=1, verbose=1)

the predictions were: array([[0.2473848 , 0.25099277, 0.251868 , 0.24975444], [0.24154082, 0.25245225, 0.25358915, 0.25241777], [0.24333884, 0.25127387, 0.25357786, 0.25180945]], dtype=float32)

How could that be if the training accuracy is 1.0?!

This is the code: def mobilenet(img_rows, img_cols, channel=1, num_classes=None):

model = MobileNet( include_top=True,weights='imagenet')

model.layers.pop()

model.outputs = [model.layers[-1].output]

model.layers[-1].outbound_nodes = []

x=Dense(num_classes, activation='softmax')(model.output)

model=Model(model.input,x)

#To set the first 8 layers to non-trainable (weights will not be updated)

for layer in model.layers[:8]:

   layer.trainable = False
model_new = Sequential()
for layer in model.layers[:-1]: # just exclude last layer from copying
    model_new.add(layer)
model=model_new
model.add(Dense(256,activation='relu',input_shape=(1000,)))

model.add(Dense(64, activation='relu'))
model.add(Dense(4,activation='softmax'))


# Learning rate is changed to 0.001
sgd = SGD(lr=1e-6,decay=1e-1,momentum=0.95, nesterov=True)
adam=Adam(lr=1e-6, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0000001, amsgrad=True)
model.compile(optimizer=adam, loss='categorical_crossentropy',metrics=['accuracy'])


# checkpoint
filepath="weights-improvement-mn-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
return model

model_mn = mobilenet(img_rows, img_cols, channel, num_classes)

model_mn.fit(X, Y,batch_size=3,epochs=50,shuffle=True,verbose=1,validation_data=(X_vall, Y_vall))

Answer 1

Chances are, you are overfitting your data, you need to closely monitor the validation accuracy, as if it diverges from the training one too much, then you enter overfitting territory.

Also, using just 3 pictures for training is just too little, your network won't be able to generalize properly with such an small input.