CNN Multi-Class Model Only Predicts 1 class for all test images

I am trying to build a CNN model to predict 42 classes. I used pre-trained models for this. I used Xception.

This is how I have imported my dataset:

train_datagen = ImageDataGenerator(rescale =1.0/255.0,
zoom_range = 0.2,
shear_range = 0.2,
horizontal_flip = True,
validation_split = 0.2)
training_data = train_datagen.flow_from_directory(train_path,
target_size = (299,299),
batch_size = 32,
class_mode = 'categorical',
subset = 'training')
validation_data = train_datagen.flow_from_directory(train_path,
target_size = (299,299),
batch_size = 32,
class_mode = 'categorical',
subset = 'validation')


I then built my model:

import keras
prior = keras.applications.Xception(include_top = False, weights = 'imagenet', input_shape = (299,299,3))
model = Sequential()
model.add(Dense(42, activation = 'softmax', name = 'Output'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])


My model performs quite well but I do not know why when I predict it on the test set, it predicts all the same class.

This is the epochs:

Epoch 1/3
2636/2636 [==============================] - ETA: 0s - loss: 1.9625 - accuracy: 0.4816
Epoch 00001: val_accuracy improved from -inf to 0.56412, saving model to xception.hdf5
2636/2636 [==============================] - 5041s 2s/step - loss: 1.9625 - accuracy: 0.4816 - val_loss: 1.6792 - val_accuracy: 0.5641
Epoch 2/3
2636/2636 [==============================] - ETA: 0s - loss: 1.4015 - accuracy: 0.6224
Epoch 00002: val_accuracy improved from 0.56412 to 0.64584, saving model to xception.hdf5
2636/2636 [==============================] - 5101s 2s/step - loss: 1.4015 - accuracy: 0.6224 - val_loss: 1.3240 - val_accuracy: 0.6458
Epoch 3/3
2636/2636 [==============================] - ETA: 0s - loss: 1.2084 - accuracy: 0.6747
Epoch 00003: val_accuracy did not improve from 0.64584
2636/2636 [==============================] - 4968s 2s/step - loss: 1.2084 - accuracy: 0.6747 - val_loss: 1.4000 - val_accuracy: 0.6373


However, when I predict its all one category:

['39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',
'39',


Any help is appreciated. I would also like to know how I can use pre-trained models for these.

• I'd recommend to check the class probabilities and potentially train your model longer. It could also be that you could benefit from a weighted loss – Valentin Calomme Jun 26 '20 at 18:50
• Checkout Focal Loss; Also you might wanna check out the code that's predicting the classes; Checkout the sampling as well; – Aditya Jun 26 '20 at 18:59
• Would you mind sharing the distribution of the targets? How many training points for each class? – Henrique Nader Jun 26 '20 at 23:55
• @ValentinCalomme The distribution of the targets are relatively similar. – eun ji Jun 27 '20 at 2:17

73 millions trainable parms

- When using Transfer learning we first freeze the base model
- Train it till you reach good accuracy
- Then unfreeze it and train for just few epochs. Keep LR small

Other probable issues -
- Add validation_split in fit method
- Suggest you add a keras.layers.GlobalAveragePooling2D after the base model and before flattening it

"setting include_top=False: this excludes the global average pooling layer and the dense output layer"

Can use this code as guidance

base_model = keras.applications.xception.Xception(weights="imagenet", include_top=False)
model = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(n_classes, activation="softmax")(model)
model = keras.Model(inputs=base_model.input, outputs=output)

for layer in base_model.layers:
layer.trainable = False

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=15, validation_data=valid_set)

for layer in base_model.layers:
layer.trainable = True

optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=5, validation_data=valid_set)


Code Ref - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition

• thank you I will try this – eun ji Jun 27 '20 at 11:05
• My model performs very poorly on my test images. I hope that you can refer to my latest post. Thank you very much. – eun ji Jul 1 '20 at 9:01

Hope you are applying the preprocessing steps on the dataset that you are using for predict. I remember getting this kind of prediction log time back and that time I think it was something to do with either not applying the same preprocessing pipeline or incorrectly doing the label map

Since you are using image generator label mapping should be easy through label_map = (train_generator.class_indices) label_map = dict((v,k) for k,v in label_map.items()) #flip k,v predictions = [label_map[k] for k in yFit] Here yFit is an array generated by model.predict()