Train/predict a classifier when the data contains more classes than we want to predict

Question

(I don't know how to phrase the title thus feel free to suggest another title).

Say I have dataset which contains images of dogs,cats,birds and other animals, and I want a classifier which only classifies dogs,cats and birds.

I could of course remove the non-used animals (e.g elephans) from the training such that the model learns what a dog,cat and bird looks like, but that would then change the distribution of dogs, cats and birds, thus I'm not sure of that.

Further more, when predicting the model would also be shown other animals (this is a toy example just to illustrate the issue thus I cannot just include the other animals and then discard the predictions of those) but should only predict cat,dog or bird.

One thought would be to set a threshold e.g if no value of the classifier is greater than, say, 0.5 then predict "other", but that would also happen if the model isn't certain if it is shown a dog, cat or bird i.e theres a difference between "other" and "not sure".

How do we ususally overcome such issues?

Answer 1

One way would be to label all the 'other' animals as a fourth class, 'other' and then train the classier on these 4 classes. The choice of this depends on the amount of data you have though of course. If the total number of 'other' far exceeds the number of items you have for dogs, cats and birds then it will add unnecessary noise.

Is there a reason you want to be able to classify the 'other' items? When you say "It would change the distribution of the dogs, cats and birds" I'm not sure I follow what you mean? Why would it be a problem in this case if you are trying to predict if an image belongs to these classes?

Another thing you might want to look into is training an open world classifier (see this paper for an example in NLP https://arxiv.org/pdf/2009.11119.pdf). The gist of it is to pair images together, and then have your labels be 1 or 0 (1 being that the labels belong to the same class, and 0 being otherwise). On inference, you would then, for a given image, pair it with K samples from all the classes, in this case dogs cats and birds, and then calculate the probability that for each pair, they belong to the same class. You would then average the probabilities for each class. If all probabilities are below a certain threshold, then you deem your test point to be 'of a class that we haven't seen in our training set', e.g. not a dog cat or a bird.