VGGnet in classification of images of new class

Question

I am using a Keras model with VGGnet as base model for image classification. Code is given below:

from keras import applications
from keras.callbacks import TensorBoard
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential, Model
from keras.layers import Dropout, Flatten, Dense
from keras import optimizers
from keras import backend as keras_backend
from keras.callbacks import TensorBoard
from tensorflow.python import debug as tf_debug

inputshape=(224,224,3)
base_model=applications.vgg16.VGG16(weights = "imagenet", include_top=True,input_shape=inputshape)
print (base_model.summary())

new_sequential=Sequential()
print(type(base_model))
for mylayer in base_model.layers:
    mylayer.trainable=False#this is done to set the weight as predefined
    new_sequential.add(mylayer)

new_sequential.layers.pop()#remove my last layer
new_sequential.add(Dense(output_dim=1,activation='sigmoid'))
new_sequential.compile(optimizer="adam",loss='binary_crossentropy',metrics=['accuracy'])

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(rescale=1./255,
                                    shear_range=0.2,
                                    zoom_range=0.2,
                                    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_data = train_datagen.flow_from_directory(directory="/home/Basic_1/dataset/training_set/",
                                                    target_size=(224, 224),
                                                    batch_size=8,
                                                    class_mode='binary')

test_validation = test_datagen.flow_from_directory("/home/Basic_1/dataset/test_set/",
                                                        target_size=(224, 224),
                                                        batch_size=8,
                                                        class_mode='binary')

cb=TensorBoard(log_dir=("/home/Basic_1"))

new_sequential.fit_generator(generator=training_data,
                            steps_per_epoch=200,
                            epochs=1000,
                            validation_data=test_validation,
                            validation_steps=5,callbacks=[cb])

from keras.models import load_model
new_sequential.save('saved_vgg.h5')
new_sequential.save_weights('saved_vgg_weights.h5')

import numpy as np
from keras.preprocessing import image

predict_image_path = "/home/Basic_1/dataset/predict/"

import glob
import os
from keras.preprocessing import image

os.getcwd()
os.chdir(predict_image_path)

images = []
images = glob.glob('*')

for imagepath in images:
    test_image=image.load_img(path=predict_image_path+'/'+imagepath,target_size=(224,224))
    test_image=image.img_to_array(test_image)
    test_image=np.expand_dims(test_image,axis=0)
    result=new_sequential.predict(test_image)
    print(result)

    if (result > 0.5):
        print("Image path {}, Obstacle    : {}".format(imagepath, result))
    else:
        print("Image path {}, Lane Follow : {}".format(imagepath, result))

Kindly help me in clarifying below doubts.

1. I have read that the VGGnet uses ImageNet for training the model. But in my case I am using a images from a custom class which is not included in the Imagenet. Will the network be able to train based on these custom classes ?

2. I see that, from ```tensorboard`` that the training seems to be normal. However the model fails during inference with new images.

3. The images in the dataset I am using to train are having 1000x540 dimension. But in this VGGnet input shape is (224, 224, 3) will the images resize during the training?

Updating the question based on the answer given:

1. Yes, I am taking an pre-trained VGGnet which is trained on ImageNet and training the model for my custom class images. (I have used the weights from pre-trained model for convolution layer and removed the FC layer from pre-trained model and used my own FC layer. Is that right what I am doing here ?

2. Yes, training loss is decreasing and accuracy is increasing. Same case with testing set as well. After running for 500 epochs I got the below accuracy and loss.

Training - Accuracy : 0.81   Loss : 0.40
Testing  - Accuracy : 0.78   Loss : 0.55

These images are not similar to ImageNet.

During Inference if I give the images from training set, model is not able to classify it correctly.

Thank you, KK

Answer 1

1. If ImageNet doesn't include the class then it won't be able to do it "out-of-the-box". But of course you can train it to do whatever you want. Essentially, you have used ImageNet for "pretraining": instead of starting from random weights, you start from weights that have been learned from some other tasks. This is called "transfer learning" in the literature. The simplest way this can be done is by (1) training an CNN (which consists of $n$ convolutional layers, followed by $m$ FC layers) on some random task (like ImageNet), (2) throwing away the $m$ FC layers and (optionally) "freezing" the weights of the $n$ conv layers, (3) adding new FC layers onto the end of network and training them on the task you are actually interested in. The reason this is done is that the conv filters learned from doing task 1 have been empirically found to be useful for general tasks; that is, the output of the pretrained conv layers is a good vector representation of image, which can be used for other tasks. The FC layers on top then learn to do whatever task you want, using that representation. So when you ask

   Will the network be able to train based on these custom classes ?

   I would say yes, assuming you are using the paradigm above (i.e., you are fine-tuning VGG), under the assumption that the conv representation is good. As for that question, see part 2.

2. Your question is missing a few potentially useful details. You wrote:

   I see that, from ```tensorboard`` that the training seems to be normal. However the model fails during inference with new images.

   Does "the training seems to be normal" just mean the training loss is decreasing nearly monotonically? Do you test on a validation set as well? Are the test (inference) images from the same set as the training images? Are these images similar to ImageNet?

   One potentially problematic case (where, for example, your pretraining might not be useful) is due to domain shift. This is when the test set is drawn from a different set/distribution than the training set. A multitude of algorithms have been designed to correct for this, called domain adaptation methods. If the training (or even pretraining) images are too different from the images at inference time, such methods are necessary for maintaining good performance.

3. The documentation for flow_from_directory say yes.

---

(Editing based on question update)

1. Yes that seems reasonable. You can try some variations: for example, allow the conv layer weights to change, but use a small learning rate (aka fine-tuning). Or use a more powerful FC multilayer network.

2. Honestly, a difference of 0.03 in accuracy between training and testing is rather minute. I would always expect the network to do at least a little worse on the test set compared to the training set. It also depends on the sizes of the sets. PAC bounds and all that :) I'm not sure what your loss is, but if it's cross-entropy-like, then you can think of the worse loss as the network being "less sure" about its responses (at least informally speaking).

   I'm not sure what you mean by During Inference if I give the images from training set, model is not able to classify it correctly. The accuracy value suggests it is doing reasonably well, and similar to the training set.

   If the images are really way different from ImageNet, then fine-tuning the conv weights may be more useful. If your dataset is also very small, looking into the domain adaptation and transfer learning literature may also be helpful.