# Neural Network Model using Transfer Learning not learning

I am a beginner in Deep Learning and working on Road Crack detection using transfer learning. I am working on binary classification with two classes , crack and no crack.

My distribution of two classes is as follows:

Cracks - 600 images

No cracks - 480 images

I have used data augmentation also :

 train_generator = train_datagen.flow(trainX, trainY, batch_size=16)

val_generator = test_datagen.flow(testX, testY, batch_size= 16)


I am using VGG16 and I have frozen the lower 4 layers like this :

vgg = vgg16.VGG16(include_top=False, weights='imagenet',
input_shape=input_shape)

output = vgg.layers[-1].output

output = keras.layers.Flatten()(output)

vgg_model = Model(vgg.input, output)

for layer in vgg_model.layers[:4]:

layer.trainable = False


After that, I added two hidden layers :

model = Sequential()
model.compile(loss='binary_crossentropy',
metrics=['accuracy'])


But after 1-2 epochs nothing seems to change, neither validation accuracy nor loss. I tried using SGD optimizer also but that also didn't help. I added more layers also but didn't have any effect on accuracy and loss.The maximum validation accuracy achieved is 62%.

I tried testing an image from my dataset, for that also model gives wrong prediction. For every test image it predicts as crack, i.e label 1.

Could someone suggest how i can improve this? Thanks!

Just take 2 images from your training data. One from class 'crack', another one from class 'not crack'. Now, check if your model can get a training accuracy of 100% which means it can overfit on the training dataset or not. If it is not able to do that, something is extremely wrong about the model.

• Yes i tried with two images, training accuracy is 100% only – Shreya Apr 4 '19 at 9:24
• Try freezing some more lower layers(the ones closer to the input). Start with freezing all the conv layers. Your dataset seems to be too small for learning that many parameters without overfitting. For checking whether the model overfits, track the training accuracies along with validation accuracies throughout the training epochs. Also, consider lowering the learning rate when you unfreeze some conv layers starting from the top(closer to the output node) so that the previously learned parameters are not completely morphed. – Sajid Ahmed Apr 4 '19 at 9:35
• Actually my model is underfitting because the training loss is always higher than validation loss and training accuracy lower than validation accuracy – Shreya Apr 4 '19 at 10:06
• Use 'categorical_crossentropy' as loss function instead of 'binary_crossentropy' since you are using 'softmax' as activation function in your output layer. I hope you have already encoded your classes in one-hot format before applying softmax....................................................... for binary_crossentropy: sigmoid activation, scalar target for categorical_crossentropy: softmax activation, one-hot encoded target – Sajid Ahmed Apr 4 '19 at 11:23
• Yes i one-hot encoded before using binary_cross entropy. I changed to categorical_cross entropy as well but there is no change in accuracy, its constant 57% – Shreya Apr 4 '19 at 13:02

Transfer learning is done by chopping off the last layer in the pre-trained network (in your case it is VGG16) and add a dense layer depending on the number of classes you need and then train the new model.

The reason why your model is not working is that you are taking the output from the last layer of the vgg16, which is activated by a softmax layer. And one cannot learn something from a softmax layer especially for transfer learning.

Rewrite your code as

from keras.models import Model
from keras.layers import Dense

X = vgg_model.layers[-1].output #will give 4096 feature vector as an output
X = Dense(256, activation ='relu')(X)
X = Dense(256, activation ='relu')(X)
X = Dense(2, activation ='softmax')(X)
newmodel = Model(vgg_model.layers[0].output,X)
newmodel.compile(loss='binary_crossentropy',
metrics=['accuracy'])

• I have already done this.Forgot to add in my code snippet here. Will edit it – Shreya Apr 6 '19 at 4:47

The problem with your code is inconsistency of the goal you are willing to operate upon. Softmax calculates the probability of individual neuron and a binary classifier contains single neuron to operate. Hence softmax is never used in binary classification and we rather use sigmoid.

So simply change the following

model = Sequential()