My question is what are potential reasons for Naive Bayes to perform well on a train set but poorly on a test set?
I am working with a variation of the 20news dataset. The dataset has documents, which are represented as "bag of words" with no metadata. My goal is to classify each document into 1 of 20 labels. My error rate on training data is 20%, but my error rate on test data is 90% (for comparison, guessing yields 95% error rate). For some reason, my classifier is predicting class 16 for almost all documents in test set. In train set this problem does not occur. Furthermore, this issue persists with different train/test splits. I'm trying to figure out what I'm doing wrong.
Here are some things I've considered:
- Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).
- Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.
- Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.
- Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.
- Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.
- Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.
Edit: Turns out I had an implementation bug. I won't detail it here as it will be unlikely to help anyone with similar issues.