# Improve accuracy of Keras multiclass image classification with pretrained VGG16 conv_base

In the moment, I'm training my first "larger" image classification model with Keras (22 classes, 2000 train samples, 500 val samples each class). I use a pretrained model (VGG16). My current model is a modified version of a sample code by F. Chollet (https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb).

Current results:

• Validation accuracy is just over 80%
• Moderate tendence of overfitting observed (viz. validation loss is not going down any further after approx. 150 epochs)
• Consequently no further improvement in the validation accuracy

I have tried a few things, such as increasing the capacity of the network and adding new/additional layers. However, by doing so, I was not able to improve the accuracy.

Since model training takes quite some time, I would like to ask for tips, how to improve the model accuracy in this setting [Note: I can not increase the number of training samples per class].

So my question(s) are:

1. Accuracy: Is a validation accuracy of about 80% in this setting (22 classes) more on the "okay" side or not? What is a reasonable accuracy (or benchmark) in such a setting? I train on relatively distinct images, such as "beach", "train", "car", "portrait" etc.
2. Model design: What would be the "next obvious thing" to improve the model accuracy? Is there a hyperparameter which I should change/target first (e.g. learning rate, batch size etc.)? Or should I (agressively) increase the capacity of the network? Or (agressively) add more layers?
3. Pretrained models: Are there (noteable) differences in the pretrained models, viz. would it be worth to switch from VGG16 to some other model?
4. Overfitting: To efficiently fight overfitting in image classification, is a i) more agressive dropout, ii) L2 regulation, or iii) batch layer normalization the best way to go?

Any hints are highly apprechiated. If there is some sample code or online resource out there, would be great. Thanks!

Here is my current model in detail:

from keras.applications import VGG16
import os, datetime, statistics
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras import models
from keras import layers
from keras import optimizers
from keras.layers.core import Dense, Dropout, Activation
from PIL import ImageFile

###############################################
# DIR with training images
base_dir = 'C:/kerasimages'
# Number training images
ntrain = 2000
# Number validation images
nval  = 500
# Batch size
batch_size = 20
# Epochs fine tuning
ep = 600
# Epochs first step
ep_first = 50
# Number of classes (for training, output layer)
nclasses = 22
###############################################

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)

def extract_features(directory, sample_count):
features = np.zeros(shape=(sample_count, 4, 4, 512))
labels = np.zeros(shape=(sample_count))
generator = datagen.flow_from_directory(
directory,
target_size=(150, 150),
batch_size=batch_size,
class_mode='binary')
i = 0
for inputs_batch, labels_batch in generator:
features_batch = conv_base.predict(inputs_batch)
features[i * batch_size : (i + 1) * batch_size] = features_batch
labels[i * batch_size : (i + 1) * batch_size] = labels_batch
i += 1
if i * batch_size >= sample_count:
break
return features, labels

# Lables and features
train_features, train_labels = extract_features(train_dir, ntrain)
validation_features, validation_labels = extract_features(validation_dir, nval)
train_labels = to_categorical(train_labels)
validation_labels = to_categorical(validation_labels)
train_features = np.reshape(train_features, (ntrain, 4 * 4 * 512))
validation_features = np.reshape(validation_features, (nval, 4 * 4 * 512))

#######################################
# Model
model = models.Sequential()
conv_base.trainable = False

#######################################
# Data generators
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(150, 150),
batch_size=batch_size,
class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
validation_dir,
target_size=(150, 150),
batch_size=batch_size,
class_mode='categorical')

# Model compile / fit
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.RMSprop(lr=2e-5),
metrics=['acc'])

history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=ep_first,
validation_data=validation_generator,
validation_steps=50,
verbose=2)

#######################################
# Fine tuning
conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
if layer.name == 'block5_conv1':
set_trainable = True
if set_trainable:
layer.trainable = True
else:
layer.trainable = False

model.compile(loss='categorical_crossentropy',
optimizer=optimizers.RMSprop(lr=1e-5),
metrics=['acc'])

history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=ep,
validation_data=validation_generator,
validation_steps=50)

• Wow, I am surprised no one could give at least a hint Sep 27 '20 at 16:48