Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am building a document classifier using Naive Bayes. There are 10 classes. My question is that :
1 Should each class contain the same number of documents for training? What if the number of training example in each class is different?
2 Does the number of classes and classification algorithm have any relation? say is there any thump rule like if there are 100 classes shall algorithm 'X' has better performance than 'Y'
Unbalanced class distributions
First, unbalanced datasets will cause your model to have a bias towards the over-represented classes. If the distribution of the classes is not very drastic then this should not cause a significant problem with any algorithm you will employ. However, as the difference between the class distribution becomes more severe you should expect to get higher false negatives for that class. Consider this, you are trying to have the model adequately identify what it means for a specific example to belong to a class. If you do not provide sufficient examples, then the model will not be able to understand the extent of the variation which exists among the examples.
If the class distribution is very different, then I would suggest anomaly detection techniques. These techniques allow you to learn the distribution of a single classification and then identify when novel examples fall within this distribution or not.
Choosing an algorithm
More classes will result in a higher dimensional output, thus contributing to the complexity of your model. For example, if you have a model which discriminates between 2 classes with a set dataset size. Then further discrimination (increasing the number of output classes) will cause the model to have higher bias. You should thus expect to see greater test error if you do not increase the size of your dataset.
If you have a set dataset X. Then you need to find the correct balance between bias and model complexity to get the optimal results. For example, a neural network based technique (highly complex) is not a good algorithm to use for a limited dataset with many output classes. However, Naive Baye's or Random Forest would be.
If the data is unbalanced then the classifier would give more weight-age that set of group which has higher number of records. So it is obvious that the outcome would be biased towards that group.
To overcome such issues we balance the data before hand.
Yes, you can use Naive Bayes Classifier, it works based on the probability. Since your problem is document classification, Naive Bayes might give you good result, as you know in most of the scenarios simple models gives best results in complex scenarios.
The other classifiers which you can try are
Random Forest
Decision Trees
SVM.
Most likely either of them would give you the desired result.
As far as I understand there is no thumb as such because an algorithm can be used in many different ways. To put in bluntly, it is trail and error, very subjective to the business problem. So, please be careful when you finalize your model.
