(I don't know how to phrase the title thus feel free to suggest another title).
Say I have dataset which contains images of dogs,cats,birds and other animals, and I want a classifier which only classifies dogs,cats and birds.
I could of course remove the non-used animals (e.g elephans) from the training such that the model learns what a dog,cat and bird looks like, but that would then change the distribution of dogs, cats and birds, thus I'm not sure of that.
Further more, when predicting the model would also be shown other animals (this is a toy example just to illustrate the issue thus I cannot just include the other animals and then discard the predictions of those) but should only predict cat,dog or bird.
One thought would be to set a threshold e.g if no value of the classifier is greater than, say, 0.5 then predict "other", but that would also happen if the model isn't certain if it is shown a dog, cat or bird i.e theres a difference between "other" and "not sure".
How do we ususally overcome such issues?