1
$\begingroup$

I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text. I use a simple naive Bayes classifier (sklearn) to try to predict a rating given a review which works reasonably well (~65% accuracy - I hope that's decent given that there are 5 classes and I'm only using a naive Bayes?)

classifier = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])
classifier.fit(x_train, y_train)
predicted = classifier.predict(x_test)
accuracy = accuracy_score(y_test, predicted) # ~65%

Anyway, what I would like to do is build a classifier that is really good at predicting if a review has the rating 4 or 5 for example. I don't really care how (bad) the classifier is doing for ratings other than 4 or 5.

My ideas were

  • modify the dataset and change all labels to 1 where the label is not 4 or 5, then fit the classifier again
  • only use the training examples that have label 4 or 5

However, both approaches do not help at all, the F-Score for classes 4 and 5 do not improve. What can I do to make a classifier better for these 2 classes only?

Improve this question
$\endgroup$
1
  • $\begingroup$ Under your rules for success, is there any upside to predicting something as having a score lower than $4?$ $\endgroup$
    – Dave
    Oct 24, 2021 at 6:06

1 Answer 1

Reset to default
0
$\begingroup$

The best approach is remove all the data that is not labeled 4 or 5.

Then rerun all the steps. Redo the train/test split and retrain the entire pipeline from scratch, including the CountVectorizer.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.