Neural Network Model using Transfer Learning not learning

Question

I am a beginner in Deep Learning and working on Road Crack detection using transfer learning. I am working on binary classification with two classes , crack and no crack.

My distribution of two classes is as follows:

Cracks - 600 images

No cracks - 480 images

I have used data augmentation also :

 train_generator = train_datagen.flow(trainX, trainY, batch_size=16)

 val_generator = test_datagen.flow(testX, testY, batch_size= 16)

I am using VGG16 and I have frozen the lower 4 layers like this :

vgg = vgg16.VGG16(include_top=False, weights='imagenet', 
                                 input_shape=input_shape)

output = vgg.layers[-1].output

output = keras.layers.Flatten()(output)

vgg_model = Model(vgg.input, output)

for layer in vgg_model.layers[:4]:

     layer.trainable = False

After that, I added two hidden layers :

model = Sequential()
model.add(vgg_model)
model.add(Dense(256, activation='relu', input_dim=input_shape))
model.add(Dense(256, activation='relu'))
model.add(Dense(2, activation='softmax')) 
model.compile(loss='binary_crossentropy',
          optimizer=Adam(lr = 1e-6),
          metrics=['accuracy'])

But after 1-2 epochs nothing seems to change, neither validation accuracy nor loss. I tried using SGD optimizer also but that also didn't help. I added more layers also but didn't have any effect on accuracy and loss.The maximum validation accuracy achieved is 62%.

I tried testing an image from my dataset, for that also model gives wrong prediction. For every test image it predicts as crack, i.e label 1.

Could someone suggest how i can improve this? Thanks!

Answer 1

Transfer learning is done by chopping off the last layer in the pre-trained network (in your case it is VGG16) and add a dense layer depending on the number of classes you need and then train the new model.

The reason why your model is not working is that you are taking the output from the last layer of the vgg16, which is activated by a softmax layer. And one cannot learn something from a softmax layer especially for transfer learning.

Rewrite your code as

from keras.models import Model
from keras.layers import Dense

X = vgg_model.layers[-1].output #will give 4096 feature vector as an output
X = Dense(256, activation ='relu')(X)
X = Dense(256, activation ='relu')(X)
X = Dense(2, activation ='softmax')(X)
newmodel = Model(vgg_model.layers[0].output,X)
newmodel.compile(loss='binary_crossentropy',
          optimizer=Adam(lr = 1e-6),
          metrics=['accuracy'])

Answer 2

Just take 2 images from your training data. One from class 'crack', another one from class 'not crack'. Now, check if your model can get a training accuracy of 100% which means it can overfit on the training dataset or not. If it is not able to do that, something is extremely wrong about the model.

Answer 3

The problem with your code is inconsistency of the goal you are willing to operate upon. Softmax calculates the probability of individual neuron and a binary classifier contains single neuron to operate. Hence softmax is never used in binary classification and we rather use sigmoid.

So simply change the following

model = Sequential()
model.add(vgg_model)
model.add(Dense(256, activation='relu', input_dim=input_shape))
model.add(Dense(256, activation='relu'))
model.add(Dense(1, activation='sigmoid')) 
model.compile(loss='binary_crossentropy',
          optimizer=Adam(lr = 1e-6),
          metrics=['accuracy'])

Talking about this model, I should say that the lower layers you have unfrozen are the layers that learn low level features like edges, so even though you have trained the lower layers it still might not have been able to detect the cracks from a higher level. As a recommendation, I would suggest you to make the higher layers of the model trainable and reduce the complexity of your model. This would improve your model.