Retraining EfficientNet on only 2 classes out of 4

Question

EfficientNet model was trained on ~3500 images for a 4-class classification: A, B, C and Neither – with accuracy of 0.985 – by someone else, not me. I'm quite new to ML. So we have this model, and it works pretty well.

As more real-world images are coming in from the users, we see more errors. Is it possible to improve the performance a bit by adding more training examples?

Collected and manually sorted 35 images of class A and 425 of class "Neither". Does it even make any sense with so small dataset?

I have put images into a folder with 4 subfolders, one per each class. Two are empty, two have images. Using Keras ImageDataGenerator flow_from_directory() method with validation/training = 0.2/0.8 ratio.

model.fit() training shows low loss and accuracy close to 1, while validation loss is high and accuracy is around 0.21 — 0.25 only.

Model performance after my "training" is totaly ruined.

Is this approach to retraining with a little set of images wrong?

How can I "add", not "replace" the model "skills" with a new added little set of images?

Maybe I should have trained a fresh model initialised with the ImageNet weights as in the initial training, and just add the new images to the set of images used, again, in the initial training?

Answer 1

The problem you describe is called transfer learning, which uses uses a pretrained model (such as EfficientNet) that is trained on an existing dataset and applies it to a different dataset. Since this method uses a pretrained model, the new dataset on which the transfer learning needs to happen doesn't need the same number of training examples compared to when the model would be trained from scratch.

In your specific case, you could look into using an existing EfficientNet model that is pretrained of a different dataset (such as ImageNet) and add one or two layers to make the model specific for your dataset (i.e. number of output neurons equal to the number of classes you are trying to classify). You can then train the pretrained model on your specific dataset, however, make sure to start by fixing the parameters of the pretrained model to make sure the model keeps the knowledge it has learnt.