Which model is better for incremental learning?

Question

I'm trying to implement face recognition. I'm planning to use some model (like DeepFace) to extract discriminative features and then use a classifier to recognize the faces. I'm confused as to which classifier to use.

My setup is as follows:
Initially, I'll have a few labelled samples and I train a classifier. Then I start getting more labelled samples and I want to update my classifier with the new data. Also, the newer samples may contain new classes (faces). So, I want my classifier to be able to learn to classify new classes as well. My question is which classifier is better suited for this task. I would also prefer to get multiple predictions with a confidence score.

1. K Nearest Neighbors algorithm: Here I can use distance as some form of confidence score.
2. As presented here, I can use SVM: Train on the initial data and retain support vectors alone. Repeat the process as and when new data comes up.

But, I have no idea if the above 2 classifiers are suited for the task at hand or how they would perform. Any inputs regarding this is greatly appreciated. I'm also open to any other classifiers as well. Thanks!

Answer 1

I will say, it's an either Or situation
You can pick one of "Incremental/Online" training Or "addition of new class".

You may do a fine-tuning approach with a Neural network by adjusting the o/p layer and training the last few layers. But this approach expects the new data to be quite similar to the training set.
KNN - Can do the online stuff but it doesn't do training. It simply calculates all the distances at the time of prediction. So, no reduction in computing. But you might have to compromise on accuracy if it is not the best
Scikit-Learn SGD Classifier can help with online training but can't support new classes

For classification, a somewhat important thing to note is that although a stateless feature extraction routine may be able to cope with new/unseen attributes, the incremental learner itself may be unable to cope with new/unseen targets classes.
SGDClassifier

Other models e.g. SVM/DT doesn't support incremental learning naturally. Though there are suggested approaches on the internet. But may not be simple. See these references.
journalofbigdata$\hspace{1cm}$ A Good SE read

I am not sure how will you figure it out if it is a new Class unless you have a separate arrangement because the model will predict it an existing Class anyway. Assuming you have a setup for that. We may try below approach -
Till the time, the prediction is within "known class" - do an Online training.
When the data is for a "New Class", do a full-data training.
With this approach, you can reduce the frequency of full-data training.

A simple Neural network can be a good candidate for both purposes. You can also get class probability with it.
Please see these useful links too
Creme
Keras + online learning

Answer 2

There is very difficult to train incremental training for up coming new classes. Once u have fixed featureset and class label for one iteration than u will have to either build new model. U can not not retrain model with different features on same model having different class label. Machine learning model is not supporting training with feature expansion. What u can do is u can train Ensemble training as described follow:

• U can train an individual model for each class that means if u have an n class than u will have n model. it would be easy for u if u have found new class in dataset. Now question u will have is that how can i validate is so for that u will have get voted for all the model and it's matric. But for that u will have each train model for 2 label like "contains class" or "not contains this class". That's how u can perform Ensemble training. U can validate model by collecting votes from each model.

Here i assumed that class label is not defined prior but it come iterativly from data.

U might look at this link. Hope this explanation will help. If u have something new idea than u can paste it here. Thank you.