High train and val results. Bad test and predict results

Question

For my thesis project I've been trying to make a CNN for some challenging data. There's four classes with the following amount of images respectively [410, 410, 269, 206] = 1,295 total.

Now I know that this is not perfect, both a small and a unbalanced dataset. I am using this tutorial as an example, resulting in this code:

# dimensions of our images.
img_width, img_height = 200, 200

top_model_weights_path = 'bottleneck_fc_model.h5'
train_data_dir = '/content/output/train'
validation_data_dir = '/content/output/val'
nb_train_samples = 760  
nb_validation_samples = 240
epochs = 30
batch_size = 10


def save_bottlebeck_features():


    # build the VGG16 network
    model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

    train_datagen = ImageDataGenerator(rescale=1. / 255,
                                      horizontal_flip=True,
                                      vertical_flip=True)

    generator = train_datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        classes = folders,
        shuffle=False)

    bottleneck_features_train = model.predict_generator(generator, nb_train_samples // batch_size)
    np.save(open('bottleneck_features_train.npy', 'wb'),
            bottleneck_features_train)

    datagen = ImageDataGenerator(rescale=1. / 255)

    generator = datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        classes = folders,
        shuffle=False)

    bottleneck_features_validation = model.predict_generator(
        generator, nb_validation_samples // batch_size)

    np.save(open('bottleneck_features_validation.npy', 'wb'),
            bottleneck_features_validation)

save_bottlebeck_features()

#def train_top_model():
train_data = np.load(open('bottleneck_features_train.npy', 'rb'))
train_labels = to_categorical(np.array(
    [0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)

print(train_data.shape)

validation_data = np.load(open('bottleneck_features_validation.npy', 'rb'))
validation_labels = to_categorical(np.array(
    [0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2)), 4)

print(validation_data.shape)

model = Sequential()
model.add(Flatten(input_shape=train_data.shape[1:]))
model.add(Dense(256, activation='relu',
                kernel_regularizer=regularizers.l2(0.0005),
                activity_regularizer=regularizers.l1(0.0005)))
model.add(Dropout(0.5))
model.add(Dense(4, activation='sigmoid'))

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(train_data, train_labels,
          epochs=epochs,
          batch_size=batch_size,
          validation_data=(validation_data, validation_labels),
          verbose=2)
model.save(top_model_weights_path)

Now my training and validation results are good:

But when I try to evaluate my model with test data which I held back, I get bad results: Found 261 images belonging to 4 classes.

test acc: 0.458248479333526

test loss: 3.629304162353702

This is just a little better then guessing.

    .......precision    recall  f1-score   support

    0kpa       0.30      0.48      0.37        82
   40kpa       0.16      0.50      0.24        42
   80kpa       0.00      0.00      0.00        82
  160kpa       0.00      0.00      0.00        55

  accuracy                         0.23      261
  macro avg     0.12      0.24     0.15      261
  weighted avg  0.12      0.23     0.15      261

I notice that the last two classes get very bad results, what could cause this? I do shuffle the data before I split it up in train/val/test. I also noticed that when i fed the model.evaluate_generator(..) with training data instead of test data it performed as bad.. that seems very odd to me.. Can anyone tell me what's going on!? Thanks in advance!

Answer 1

I figured it out.

First: I was making the labels the wrong way.

Instead of this:

train_labels = to_categorical(np.array(
[0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)

do this:

train_labels = to_categorical(training_generator.classes, 4)

The first snippet wrongly generated a 4-ary array, where only the first and second element got a 1 assigned.

Second:

I found out my method of evaluation was wrong. I tried to use model.evaluate_generator which yielded the wrong results. I think it was because I tried to feed the model raw images, instead of creating bottleneck features from the raw test images and then feeding them to the model.

so instead of this:

from keras.models import load_model
conv_base = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(conv_base)
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1./255)


test_generator = test_datagen.flow_from_directory(
        '/content/output/test',
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical')

test_loss, test_acc = model.evaluate_generator(test_generator, 216 // batch_size)
print('test acc:', test_acc)
print('test loss:', test_loss)

I did this (Which worked):

from keras.models import load_model
model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

bottleneck_features_test = model.predict_generator(
    test_generator, 261 // batch_size)

loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    '/content/output/test',
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical',
    classes = folders,
    shuffle=False)

bottleneck_features_test = conv_base.predict_generator(
    test_generator, 261 // batch_size)

test_labels = to_categorical(test_generator.classes, 4)

(test_eval_loss, test_eval_accuracy) = model.evaluate(  
   bottleneck_features_test[:256], test_labels[:256], batch_size=batch_size, verbose=1)

print("[INFO] test accuracy: {:.2f}%".format(test_eval_accuracy * 100))  
print("[INFO] test Loss: {}".format(test_eval_loss)) 

I hope this helps someone who's struggling with the same!

Answer 2

Somehow, the image generator of Keras works well when combined with fit() or fit_generator() function, but fails miserably when combined with predict_generator() or the predict() function.

When using Plaid-ML Keras back-end for AMD processor, I would rather loop through all test images one-by-one and get the prediction for each image in each iteration.

import os
from PIL import Image
import keras
import numpy

# code for creating dan training model is not included

print("Prediction result:")
dir = "/path/to/test/images"
files = os.listdir(dir)
correct = 0
total = 0
#dictionary to label all animal category class.
classes = {
    0:'This is Cat',
    1:'This is Dog',
}
for file_name in files:
    total += 1
    image = Image.open(dir + "/" + file_name).convert('RGB')
    image = image.resize((100,100))
    image = numpy.expand_dims(image, axis=0)
    image = numpy.array(image)
    image = image/255
    pred = model.predict_classes([image])[0]
    animals_category = classes[pred]
    if ("cat" in file_name) and ("cat" in sign):
        print(correct,". ", file_name, animals_category)
        correct+=1
    elif ("dog" in file_name) and ("dog" in animals_category):
        print(correct,". ", file_name, animals_category)
        correct+=1
print("accuracy: ", (correct/total))
```