Optimize F-Score only for certain classes, disregard other classes

I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text. I use a simple naive Bayes classifier (sklearn) to try to predict a rating given a review which works reasonably well (~65% accuracy - I hope that's decent given that there are 5 classes and I'm only using a naive Bayes?)

classifier = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])
classifier.fit(x_train, y_train)
predicted = classifier.predict(x_test)
accuracy = accuracy_score(y_test, predicted) # ~65%


Anyway, what I would like to do is build a classifier that is really good at predicting if a review has the rating 4 or 5 for example. I don't really care how (bad) the classifier is doing for ratings other than 4 or 5.

My ideas were

• modify the dataset and change all labels to 1 where the label is not 4 or 5, then fit the classifier again
• only use the training examples that have label 4 or 5

However, both approaches do not help at all, the F-Score for classes 4 and 5 do not improve. What can I do to make a classifier better for these 2 classes only?

• Under your rules for success, is there any upside to predicting something as having a score lower than $4?$
– Dave
Oct 24, 2021 at 6:06

1 Answer

The best approach is remove all the data that is not labeled 4 or 5.

Then rerun all the steps. Redo the train/test split and retrain the entire pipeline from scratch, including the CountVectorizer.