How can we scale up the number of classes for deep learning after training a model?

Question

I have trained a deep learning model for a face recognition application. The model has been trained on the existing subjects (or call it classes) in the gallery. Now consider the real world problem, where the gallery size is dynamic as new subjects are to be enrolled, how can deep learning solve those type of problems? I have tried transfer learning, are there any other techniques?

Answer 1

Face recognition from images is still an open area of research. Many different techniques have been tried.

One approach is not to directly learn the identities of the people as classes, but to learn a multi-dimensional embedding for images of faces (e.g. 128 dimension vector) and train the network to make this embedding value closer for two different images of the same person than for two images of different people*.

You can use the network to generate feature vectors for each person who is registered, and store those values. Later when you want to identify someone, find the closest one (according to norm of the difference) and if the distance is less than a certain cutoff you can consider it to be a match.

To perform the training for the embeddings with this goal in mind, you can use a loss function called triplet loss, which is trained on 3 forward passes at a time. The network calculates feature vectors for an anchor image, a positive match (i.e. another image of the same face, so you need at least 2 for each person, and they should be taken from different contexts) and a negative match. Then it measures whether the distance between features from same person is less than distance between different people. The loss function to express that looks like:

$$\text{max}(0, |A - P|^2 - |A - N|^2 + \alpha)$$

. . . where $A$ is feature vector for the anchor image, $P$ for positive match image, and $N$ for negative match image. The parameter $\alpha$ is a hyper-parameter which creates a "margin" that you reward the model for maintaining between different identities. The idea being that if an image of different person is clearly further away in the feature space than a cutoff, the loss on that training example is zero. The model does not get rewarded for very large differences, just having "consistently enough distance".

The tricky thing with triplet loss is picking which samples to use for training. Purely random samples could make the network's task too easy. You need it to focus a bit on the more difficult cases to prevent mistaken identity. One way of doing this is to pick preferentially from closer faces, e.g. using k-means clusters and picking negative cases within own cluster.

The model can be trained on large numbers of faces, not necessarily the ones you will later want to classify. The point is to learn a feature vector that emphasises important differences between faces (and ideally ignores other variations such as lighting, background, pose). It's possible to start with any image classifier and adapt it for this task, including your original face classifier (just remove the final classifier layer and work with one of the later fully-connected layers as your output).

How can we scale up the number of classes for deep learning after training a model?

This approach may not work well in all cases covered by your question title. I have recommended it for face recognition based on a Coursera course on CNNs where a few lectures and coding assignment are about face recognition specifically.

In some cases you may be better off re-building and re-training the model with new training set data, augmented by examples of your new class - you might start by fine-tuning only the last few layers.

---

* In fact this will be the case in many CNN classifiers, but is more likely to be reliably linked to a person's identity if you have trained the network explicitly for distinguishing between identities in general (as opposed to just classify specific identities). The triplet loss approach is one way to train for useful representations, and makes the goal of disambiguating identity explicit.