5
$\begingroup$

I am solving an image classification problem. However some photos may not belong to any category, and I'd like not to give any fake information, rather to capture this situation. What are the ways to do it. One idea I have in mind is to give an extra category "OTHER" but then I need to have some data for this category which would be difficult to gather as it's too general.

$\endgroup$
2
  • $\begingroup$ There are billions possible categories and subcategories... So you aren't doing anything wrong when adding an Unknown class until and unless you want to train on all.... That's the safest and one of the commonly used style... Or if your preds are inconclusive, then vote for that but that's fake.. $\endgroup$ – Aditya Sep 21 '18 at 16:06
  • $\begingroup$ You have chosen a good path. Try to do data augmentation. $\endgroup$ – Media Sep 21 '18 at 18:16
3
$\begingroup$

This question is a very common question. How to handle samples belong to categories that are not included in the data set. There is no optimal way to handle this:

Creating a separate class is possible but not recommended. First, you need images to start with, second this class will have a very large variation within the class, a property that is not favorable which makes designing a powerful classifier to handle the other classes more challenging and usually results in a low quality classifier.

I prefer to handle this by measuring the confidence. If you are expecting to have anomalies/ outliers, make sure to use classifier that gives both prediction and probabilities ( confidence).

CNN is good enough , you can only accept predictions if the confidence is a higher than a threshold, or the difference between the highest confidence ( label) and the next confidence (label) is higher than a threshold. Most of the probability based classifies can give you confidence values.

Some other classifiers can be changed to give confidence, for example, you can analyze the training data used to give a label for a leave in decision tree, these samples belonging to the same leave can be clustered to get some centroids. In prediction you can predict the label and measure the shortest distance between your target ( testing sample) and each of the centroids belonging to the same leave you end with, that shortest distance can be used as a measure of confidence and can be compared to a threshold to accept / reject the prediction.

Please note that this problem is still under research and there is no clear answer to it. It needs very deep understanding to each classifier to modify it / or add to it to measure confidence and to understand what that measure means

$\endgroup$
3
  • $\begingroup$ yeah great answer! $\endgroup$ – Francesco Pegoraro Sep 24 '18 at 20:45
  • $\begingroup$ Thanks! Can you give an example of a classifier, that gives its confidence directly? I only know about measures of confidence/uncertainty build up on predictions such as entropy of predictions. Anything else you could suggest? $\endgroup$ – kmichael08 Sep 25 '18 at 7:40
  • $\begingroup$ Updated the answer $\endgroup$ – Bashar Haddad Sep 25 '18 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.