Pre-trained CNN model makes Poor Predictions on Test Images Dataset

Question

I have tried using several a pretrained models (MobileNet) for multiclass predictions. There are 42 classes and the distributions of the images are even across the 42 classes.

This is my code:

base_model=MobileNet(weights='imagenet',include_top=False,input_shape = (224,224,3)) #imports the mobilenet model and discards the last 1000 neuron layer.
x=base_model.output
x=GlobalAveragePooling2D()(x)
x=Dense(512,activation='relu')(x) #we add dense layers so that the model can learn more complex functions and classify for better results.
preds=Dense(42,activation='softmax')(x) #final layer with softmax activation
model=Model(inputs=base_model.input,outputs=preds)
for layer in base_model.layers[:20]:
    layer.trainable=False
for layer in base_model.layers[20:]:
    layer.trainable=True

I have freezed and unfreezed some of the trainable layers.

I let it run for 5 epochs:

Epoch 1/5
1318/1318 [==============================] - 3604s 3s/step - loss: 1.5493 - accuracy: 0.5796 - val_loss: 1.7180 - val_accuracy: 0.5361
Epoch 2/5
1318/1318 [==============================] - 3272s 2s/step - loss: 1.2174 - accuracy: 0.6641 - val_loss: 1.7562 - val_accuracy: 0.5372
Epoch 3/5
1318/1318 [==============================] - 3233s 2s/step - loss: 1.0853 - accuracy: 0.6981 - val_loss: 1.2993 - val_accuracy: 0.6498
Epoch 4/5
1318/1318 [==============================] - 3223s 2s/step - loss: 0.9918 - accuracy: 0.7224 - val_loss: 1.3455 - val_accuracy: 0.6382
Epoch 5/5
1318/1318 [==============================] - 3310s 3s/step - loss: 0.9153 - accuracy: 0.7413 - val_loss: 1.2375 - val_accuracy: 0.6660

The accuracy I got is relatively good. I have tried with other pretrained models like Xception too and the accuracies were pretty good. However when I use my model to predict on the test data, and upload my submission to Kaggle, my predictions are very bad.

I am wondering if I have imported the test images and made predictions correctly? Here is the code:

import cv2
pred_images = []
filename= []

for image_file in os.listdir('./test/test'): #Extracting the file name of the image from Class Label folder
    filename.append(image_file)
    image = cv2.imread('./test/test'+r'/'+image_file) #Reading the image (OpenCV)
    image = cv2.resize(image,(224,224)) #Resize the image, Some images are different sizes. (Resizing is very Important)
    pred_images.append(image)

category = []
for i in range(len(pred_images)):
    pred_image = np.array([pred_images[i]])
    pred = model.predict(pred_image)
    cat = np.argmax(pred, axis = 1)
    category.extend(cat)

I get a data frame looking like this, with a lot from class 38 and 20. Is there something wrong?

final = pd.DataFrame(list(zip(filename, category)), columns = ['filename', 'category'])
final.head()

    filename                               category
0   c94de2fa9b06d67848f648e33a43475c.jpg    38
1   bbb7a2da148488bb878727556aa5914c.jpg    38
2   606256bdf3636d280bfdc3def33a57e7.jpg    38
3   7f741619b952876e7c7c419a0de1ed60.jpg    20
4   05c7592b31ceb8e14d4faa30fa21794c.jpg    20

Answer 1

Kaggle's Dataset is really noisy and it is really tough to get very good result with simple models. You have to do lots of preproccesing ,Data Augmentations and larger network for many more epoch to get a good result. Things you can do :

1. Go to notebook section of your competetion and look out for some preprocessing and augmentations and apply them.
2. Use some regularizer like dropout, mixup,cutmix and many more.

3.Train your model for more epoch

4. Train a larger model.