I have trained my classifier on pictures with a mixture of several classes on each picture, e.g. A-F. The classifier is able to (nearly) correctly segment those classes on the images.

Now I got more data with pictures showing class G. To minimize my work, I only labeled class G on the images and left the rest out (Invalid).

Two questions for my training arise:

  1. If there are no examples of class G in my first dataset (because it could have been a subclass of A-F), how do I train it correctly? Suggested Solution: I can add my new data to the old one, but I have to tell the CNN when there is an image with class G, because I have to change the error measurement of my CNN somehow. But how?
  2. If G was included in my first dataset (assume G is 'police car' and B is 'car', but some images showed police car), how do I train in this case correctly?

2 Answers 2


Transfer learning is what you want to need.

Checkout this note from CS231n, it provides some general advices on transfer learning and model fine-tuning.

In the meanwhile, this blog from keras shows how to use pre-trained VGG16 network to classify dogs&cats.

  • $\begingroup$ I know that I have to finetune - but I am interested in how to adapt the new classes. Any more specific answers on how to model it exactly? $\endgroup$
    – N8_Coder
    Commented Feb 20, 2017 at 6:27
  • $\begingroup$ Does this gist help? $\endgroup$
    – Icyblade
    Commented Feb 21, 2017 at 6:23

To adapt your classifier to the new class, you need to modify the architecture of your neural network to accommodate the new class, and retrain the model on the combined dataset (i.e., the original dataset plus the new labeled data containing only class G).

One approach to adapting your neural network is to add a new output layer to the existing model to classify the new class. This new output layer will have only one neuron, corresponding to the binary classification of class G vs. the "Invalid" class. You can initialize the weights of the new output layer randomly and freeze the weights of the pre-existing layers, so that the pre-existing features learned from the original dataset are retained and only the weights of the new output layer are updated during training.

Another approach is to fine-tune the entire neural network on the combined dataset, including the weights of the pre-existing layers. In this case, you would initialize the weights of the pre-existing layers with the pre-trained weights from the original model, and the weights of the new output layer would be randomly initialized. During training, all the weights in the model are updated to minimize the loss on the combined dataset.

To handle the class imbalance in the combined dataset, you can use a weighted loss function that assigns higher weights to misclassified examples from the new class, or use data augmentation techniques to generate synthetic examples of the new class. Additionally, you can employ techniques such as dropout and early stopping to prevent overfitting.

After training, you can evaluate the performance of the adapted classifier on a validation set and fine-tune the hyperparameters as necessary to optimize the performance.

In the first case where your initial dataset doesn't include class G, you can use transfer learning to leverage the pre-trained features of your existing classifier. You can fine-tune your pre-trained model on the new data with class G by replacing the last layer(s) of your classifier and training only those layers with the new data.

To change the error measurement of your CNN when training with the new data, you can modify the loss function to include a new term that penalizes misclassification of class G. One way to achieve this is by adding a weighted cross-entropy loss, where the weight for class G is increased to reflect the class imbalance in your new dataset. Another option is to use a focal loss, which is a variant of the cross-entropy loss that focuses more on hard-to-classify examples. By tuning the parameters of the loss function, you can control how much emphasis the CNN places on class G during training.

In the second case where your initial dataset already includes class G, you can train your classifier in a similar manner as before, by adding the new data to the existing dataset and modifying the loss function as necessary. However, you should be careful to avoid overfitting to class G, since it is now represented in both datasets. To do so, you can use techniques such as data augmentation, dropout, and early stopping to prevent the model from memorizing the new examples and instead encourage it to learn more generalizable features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.