Val Accuracy not increasing at all even through training loss is decreasing

Question

I am training a model for image classification, my training accuracy is increasing and training loss is also decreasing but validation accuracy remains constant.

Here is my code:

from keras.applications.vgg19 import VGG19 model= VGG19(include_top=False, weights='imagenet', 
        input_tensor=None, input_shape=(224,224,3), pooling=None, classes=1000) x=model.output

x=Conv2D(filters=1024,kernel_size=2)(x)
x=MaxPooling2D()(x) 
x=Flatten()(x) 
x=Dense(1024,activation='relu')(x) 
x=BatchNormalization(axis=1)(x) 
x=Dropout(0.8)(x) 
x=Dense(64,activation='relu')(x) 
x=Dense(4,activation='softmax')(x)

model = Model(inputs=model.input,outputs=x)

for layer in model.layers[:12]:
   layer.trainable = False

for layer in model.layers[12:]:
   layer.trainable=True


opt = Adam(lr=0.0001, decay=1e-6) 
model.compile(loss='sparse_categorical_crossentropy', 
           optimizer=opt, metrics=['accuracy'])

checkpoint_path="/content/drive/My Drive/Model/model_vgg19_6.h5"

checkpoint = ModelCheckpoint(checkpoint_path, monitor="val_acc", mode="max", 
   save_best_only = True,verbose=1)

reduce_lr = ReduceLROnPlateau(monitor = 'val_acc', mode="max", factor = 0.7, 
     patience = 5, verbose = 1, min_delta =0.00001)

earlystop = EarlyStopping(monitor = 'val_acc', mode="max", min_delta = 0, patience = 30, verbose = 1, 
            restore_best_weights = True)

callbacks = [reduce_lr,checkpoint]

model.fit_generator(aug_train, steps_per_epoch=int((len(data_x)/128)+1), validation_data= 
                (val_x,val_y), validation_steps=int((len(val_x)/128)+1), workers=-1, 
                use_multiprocessing=True, shuffle=True, epochs=300, callbacks=callbacks )

I am getting a constant val_acc of 0.24541

Answer 1

You start with a VGG net that is pre-trained on ImageNet - this likely means the weights are not going to change a lot (without further modifications or drastically increasing the learning rate, for example).

If you are expecting the performance to increase on a pre-trained network, you are performing fine-tuning. There is a section on fine-tuning the Keras implementation of the InceptionV3 network, but the principals are the same: you should freeze some of the earlier feature-extraction layers, leaving only some of the final layers marked as trainable. This is the example given in the docs, where they add new layers to the base model, train only those layers for a while, then additionally unfreeze some of the base model.

# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)   # <--- leave out final layers!
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
    layer.trainable = False

-> run training for a few epochs

The some time later, unfreeze the part of the base model

# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 249 layers and unfreeze the rest:
for layer in model.layers[:249]:
   layer.trainable = False
for layer in model.layers[249:]:
   layer.trainable = True

Please looked at the full documentation for more details.