CNN model low accuracy

Question

I have 1299 images in 4 classes (374/269/284/372). I want to use the VGG19 model, add a dense layer at the top and fine-tune it with my images. As I only have 1299 images, I also want to use data augmentation.

Here is the code (it is not exactly the code I used everytime, as I tried several things. When I did not use preprocess_input, I did X = X/255.) :

drive.mount('/content/gdrive')

DATADIR = '/content/gdrive/My Drive/'
CATEGORIES = ['A','B','C','D']
training_data = []

def create_training_data():
    for category in CATEGORIES:
        path = os.path.join(DATADIR,category)
        class_num = CATEGORIES.index(category)
        for img in tqdm(os.listdir(path)):
            img_array = cv2.imread(os.path.join(path,img))
            img_array_RGB = cv2.cvtColor(img_array, cv2.COLOR_BGR2RGB)
            new_array = cv2.resize(img_array_RGB, (224,224))
            training_data.append([new_array,class_num])

create_training_data()

X = []
Y = []

for features, label, in training_data:
    X.append(features)
    Y.append(label)
    
X = np.array(X)
Y = np.array(Y)

IMG_SIZE=224
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 3)

Y_class = Y
Y = to_categorical(Y, num_classes=4)


X_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size = 0.3, random_state=0, stratify=Y)

Y_train_int, Y_val_int = [np.where(r==1)[0][0] for r in Y_train], [np.where(r==1)[0][0] for r in Y_val]

from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
        horizontal_flip=True,
        vertical_flip=True,
        brightness_range=[0.2,0.6],
        fill_mode='wrap')

datagen.fit(X_train)

vgg_model = VGG19(weights='imagenet', include_top=False)

X = preprocess_input(X) 

x = vgg_model.output
x = GlobalAveragePooling2D()(x)

x = Dense(512, activation='relu')(x)
x = Dropout(0.3)(x)

predictions = Dense(4, activation='softmax')(x)

model = Model(inputs=vgg_model.input, outputs=predictions)

for layer in vgg_model.layers:
    layer.trainable = False

layer_num = len(model.layers)
for layer in model.layers[:21]:
    layer.trainable = False

for layer in model.layers[21:]:
    layer.trainable = True



#class_weights = class_weight.compute_class_weight('balanced',
#                                                 np.unique(Y_train_int),
#                                                 Y_train_int)

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(datagen.flow(X_train,Y_train, batch_size=64), epochs=30, batch_size=128, shuffle=True, validation_data = (X_val,Y_val))

However this does not produce relevant results. I tried several modifications (changing the learning rate, batch size, loss as sparse_categorical_crossentropy by changing from one hot encoding to integer, adding weights to classes, and I also tried to build a CNN from scratch with keras, but nothing seems to give better results (predicting the class with the most images)

Here is an example of plots : https://i.stack.imgur.com/KODAr.png

I don't really know what else I could try, but I really think a good model could classify these images better than this.

Edit : I changed some parameters :

• Data augmentation (I deleted the brightness)
• batch_size = 32 for the data augmentation and 64 for the model
• added weights to the classes
• added x = Dense(256, activation='relu')(x) after x = Dense(512, activation='relu')(x)

And now the learning curve and confusion matrix look like this (only 10 epochs) : https://i.stack.imgur.com/IsJTh.png

The learning curve is really fluctuating and the model doesn't seem to learn, but the accuracy is higher than before. I don't know what else to do now.

Answer 1

So, the question is asking for advice on what to do next, given the model's performance. Firstly, in terms of confusion matrices, it is often useful to display proportions, rather than number of examples. It makes it easier for us to gauge where the error is occurring.

Looking at the most recent confusion matrix, it is clear that a lot of the examples are classified into classes 2 and 3. This could be due to a couple of things, potentially class imbalance, especially since the class weights is commented out in the code. Also, looking that the loss over epochs, it looks like the loss could continue to decline if you gave the model more epochs (unless you put a callback on it to prevent it from continuing training, which doesn't look to be the case).

Try that out and edit the post accordingly with the updated results!