I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text. I use a simple naive Bayes classifier (sklearn) to try to predict a rating given a review which works reasonably well (~65% accuracy - I hope that's decent given that there are 5 classes and I'm only using a naive Bayes?)
classifier = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())]) classifier.fit(x_train, y_train) predicted = classifier.predict(x_test) accuracy = accuracy_score(y_test, predicted) # ~65%
Anyway, what I would like to do is build a classifier that is really good at predicting if a review has the rating 4 or 5 for example. I don't really care how (bad) the classifier is doing for ratings other than 4 or 5.
My ideas were
- modify the dataset and change all labels to 1 where the label is not 4 or 5, then fit the classifier again
- only use the training examples that have label 4 or 5
However, both approaches do not help at all, the F-Score for classes 4 and 5 do not improve. What can I do to make a classifier better for these 2 classes only?