I am solving an image classification problem. However some photos may not belong to any category, and I'd like not to give any fake information, rather to capture this situation. What are the ways to do it. One idea I have in mind is to give an extra category "OTHER" but then I need to have some data for this category which would be difficult to gather as it's too general.
This question is a very common question. How to handle samples belong to categories that are not included in the data set. There is no optimal way to handle this:
Creating a separate class is possible but not recommended. First, you need images to start with, second this class will have a very large variation within the class, a property that is not favorable which makes designing a powerful classifier to handle the other classes more challenging and usually results in a low quality classifier.
I prefer to handle this by measuring the confidence. If you are expecting to have anomalies/ outliers, make sure to use classifier that gives both prediction and probabilities ( confidence).
CNN is good enough , you can only accept predictions if the confidence is a higher than a threshold, or the difference between the highest confidence ( label) and the next confidence (label) is higher than a threshold. Most of the probability based classifies can give you confidence values.
Some other classifiers can be changed to give confidence, for example, you can analyze the training data used to give a label for a leave in decision tree, these samples belonging to the same leave can be clustered to get some centroids. In prediction you can predict the label and measure the shortest distance between your target ( testing sample) and each of the centroids belonging to the same leave you end with, that shortest distance can be used as a measure of confidence and can be compared to a threshold to accept / reject the prediction.
Please note that this problem is still under research and there is no clear answer to it. It needs very deep understanding to each classifier to modify it / or add to it to measure confidence and to understand what that measure means