# Good test accuracy but poor confusion matrix results

Ive trained a model to classify 4 types of eye diseases using MobileNet as the pretrained model. I achieved a test accuracy of 94%, but when I look at the confusion matrix, it seems like it isn't doing so well. Loss is relatively low on training, validation, and testing. Any suggestions on where I went wrong or if im missing something conceptually?

Image_height = 224
Image_width = 224
val_split = 0.20
batches_size = 16
lr = 0.0005
spe = 220
vs = 32
epoch = 6

# Getting the file of the training set and testing set
train_folder = "/content/drive/My Drive/Research/train"
test_folder = "/content/drive/My Drive/Research/test"

#Creating batches
train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet.preprocess_input,validation_split=val_split) \
.flow_from_directory(directory=train_folder, target_size=(Image_height,Image_width), classes=['CNV','DME','DRUSEN','NORMAL'], batch_size=batches_size,class_mode="categorical",
subset="training")
validation_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet.preprocess_input,validation_split=val_split) \
.flow_from_directory(directory=train_folder, target_size=(Image_height,Image_width), classes=['CNV','DME','DRUSEN','NORMAL'], batch_size=batches_size,class_mode="categorical",
subset="validation")
test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet.preprocess_input) \
.flow_from_directory(test_folder, target_size=(Image_height,Image_width),
classes=['CNV','DME','DRUSEN','NORMAL'], batch_size=batches_size,class_mode="categorical")

mobile = tf.keras.applications.mobilenet.MobileNet(include_top=False,
input_shape=(224, 224,3),
pooling='max', weights='imagenet',
alpha=1, depth_multiplier=1,dropout=.5)
x=mobile.layers[-1].output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
predictions=Dense (4, activation='softmax')(x)
model = Model(inputs=mobile.input, outputs=predictions)
for layer in model.layers:
layer.trainable=True
checkpoint=tf.keras.callbacks.ModelCheckpoint(filepath="/content/drive/My Drive/Research/ModelCheckpoint", monitor='val_loss', verbose=0, save_best_only=True,
save_weights_only=False, mode='auto', save_freq='epoch', options=None)
lr_adjust=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1, verbose=0, mode="auto",
min_delta=0.00001,  cooldown=0,  min_lr=0)

model.fit(train_batches, steps_per_epoch=spe,
validation_data=validation_batches,validation_steps=vs, epochs=epoch)

# Predict the accuracy on the Test set
acc = model.evaluate_generator(test_batches, steps=len(test_batches), verbose=1)
print("Model Accuracy on Test Data", acc[1]*100)

y = []
for x in range(0,len(test_batches)):
for i in range(0,len(test_batches[x][1])):
#print(test_batches[0][1][i])
y.append(np.argmax(test_batches[x][1][i]))
print(len(y))

con_mat = tf.math.confusion_matrix(labels=y, predictions=np.argmax(predictions,axis=1)).numpy()
print(con_mat)


Training/Validation

Epoch 1/6
220/220 [==============================] - 2952s 13s/step - loss: 0.5842 - accuracy: 0.7912 - val_loss: 0.7926 - val_accuracy: 0.7988
Epoch 2/6
220/220 [==============================] - 2736s 12s/step - loss: 0.4041 - accuracy: 0.8723 - val_loss: 0.3094 - val_accuracy: 0.9023
Epoch 3/6
220/220 [==============================] - 2635s 12s/step - loss: 0.3718 - accuracy: 0.8804 - val_loss: 0.3871 - val_accuracy: 0.8906
Epoch 4/6
220/220 [==============================] - 2517s 11s/step - loss: 0.2904 - accuracy: 0.8980 - val_loss: 0.2863 - val_accuracy: 0.9160
Epoch 5/6
220/220 [==============================] - 2364s 11s/step - loss: 0.2779 - accuracy: 0.9057 - val_loss: 0.3500 - val_accuracy: 0.9238
Epoch 6/6
220/220 [==============================] - 2241s 10s/step - loss: 0.2839 - accuracy: 0.9068 - val_loss: 0.2202 - val_accuracy: 0.9355
<tensorflow.python.keras.callbacks.History at 0x7f6f8a59eb70>


Testing

WARNING:tensorflow:From <ipython-input-12-d213edec98d3>:2: Model.evaluate_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.evaluate, which supports generators.
63/63 [==============================] - 837s 13s/step - loss: 0.1519 - accuracy: 0.9410
Model Accuracy on Test Data 94.0999984741211


Confusion Matrix

[[70 62 57 61]
[82 61 41 66]
[74 69 49 58]
[77 60 48 65]]

• Should be a bug in the code. If your confusion matrix is correct you should have an accuracy of the sum of the diagonal of the confusion matrix divided by the sum of all it's entries. Aug 14 '20 at 22:36
• Have you checked your outputs? Does each row sum to 1? Aug 14 '20 at 22:37
• @TimvonKänel Just checked, all of the outputs for each row sum to 1 Aug 15 '20 at 2:28
• I actually think your confusion matrix is just wrong. Have you checked if the amount of elements in the matrix equals your test_set size? Aug 15 '20 at 8:07
• You have only 220 Train and 63 Test data but your confusion matrix has so many numbers ~1000. Please check the code of your confusion matrix Aug 15 '20 at 16:33

The problem is that ImageDataGenerator.flow_from_directory shuffle argument is True by default, so you must set it False and get the data in alphanumeric order. This causes consistent label assignment to images and also because of data is totally unseen in training phase it won't causes any issue on generalization of model.


test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet.preprocess_input) \
.flow_from_directory(
test_folder,
target_size=(Image_height, Image_width),
classes=['CNV', 'DME', 'DRUSEN', 'NORMAL'],
batch_size=batches_size,
class_mode="categorical",
shuffle=False  # here is the change 🌟
)


Similar question has been answered here

Im not an expert in Tensorflow, but you pass the argument "predictions" to tf.math.confusion_matrix, which should be a 1D Tensor. However, I don't see you creating it using your test_set.

• I'll check this out thanks! Aug 15 '20 at 17:19