I have a classification problem, where I initially started with 100+ class labels. My intuition told me this is too many labels for a model to predict with good accuracy. I thought grouping the labels together will help the situation and it did, there is now only 19 class labels. Is 19 still too many class labels? What are the typical signs that there are too many labels for a model to predict and how can this be addressed, if there is too many class labels?
A class taxonomy should:
- Serve the business needs
- Be learnable
There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be able to server better the business needs. However, for a large taxonomy the classifier will have to model more complex rules, will have less samples for each case and the boundary samples will have more influence.
Check the number of samples per label in your dataset. A class label with not enough samples will be hard to learn. If you have many class label, the ratio of some of them will also be low, introducing an imbalance learning problem which is also harder.
Using a cost matrix, you can evaluate the extra cost due to merging some classes. merge classes when the cost on not distinguishing among them is low.
Building a taxonomy of class might can help working in many resolutions. Take for example animals, you can create a taxonomy of animals -> reptiles -> crocodiles and then try to distinguish reptiles from other animals or crocodiles from other reptiles.
In many some cases you might get many class labels due to a Cartesian product of some dimensions. Using the animals examples again, you might be interested in dimension like "is_flying?", "is_carnivore?" and learn them instead of the animal classes. These dimensions will probably be more balanced and can be learn with a certain degree on independence (reusing samples). The down side is the a flying carnivore will be a bird of prey and not necessarily a falcon.
There is no clear cut answer for this. It depends on the size of your dataset, the computing power you have at your disposal and the accuracy you get compared to the number of classes.
Some signs or rules of a thumb are:
The smaller the dataset (both in data points and feature space dimensionality) the less class labels you can manage successfully.
How fast can you train your model for each class? If adding that for all the classes results in acceptable time you can keep them. This applies to cases where you are training a model to predict classes in a one vs all setting. K-means clustering on the other will not be significantly affected by increasing the number of classes.
Can you get good accuracy from your dataset with all those classes? If not reduce them to decrease the complexity of your problem.
For each case the balance of the above factors will give a different question.
Last but not least the number of classes to keep is influenced by the specific domain that gave you the dataset. You should use domain knowledge in addition to machine learning methods to assess whether you have too many classes or not.