1
$\begingroup$

I am creating my own implementation of a Naïve Bayes classifier. While it’s behaviour and functionalities are clear to me, my concerns are on the nature of the training and testing data.

I acquired several sets of product reviews from Amazon. The first thing I do is parsing them, that is, taking the rating (1 to 5 stars) and the text, which I parse with a regex to only contain alphabetical lowercase characters and spaces. Next, I convert ratings to polar values, so 1 and 2 stars become “-“ and 4 and 5 stars become “+”. I’m intentionally skipping reviews with 3 stars; could this be an issue?

Here come my real concerns. When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews (such as 7 positive and 7 negative reviews for training and 3 positive and 3 negative reviews for testing)? Right now I’m acquiring as many positive as negative reviews from the chosen set, but I’m wondering if that should be the case. For instance, if a set contains 7 positive reviews and 4 negative ones, I discard 3 positive reviews to equate them.

Furthermore, I observed that negative reviews tend to contain longer texts on average. So, if I’m using an equal number of positive and negative reviews, but they differ on average text length, would this have an impact on the way my classifier tries to predict?

$\endgroup$

1 Answer 1

0
$\begingroup$

I’m intentionally skipping reviews with 3 stars; could this be an issue?

It's not a problem, it's just your definition of your question. It assumes rate 3 as neutral and blow/above that as negative/positive. One may say I assume every rate is positive so each number is a level of positiveness then rate 3 works for him. The main point is that you get the answer for the question you define so be careful about what you want from your classifier to set up your question correctly.

When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews?

Nice question. Before specifically taking Naive Bayes into account, it is a general machine learning problem when the population of classes are imbalanced. If this is the case then better to balance them, not only in train/test split but also during train itself as the dominating class will bias your result. If the classes are not that imbalanced then you can split things randomly and it's fine. Regarding Bayes Classifier itself the balance inside training set should be more important as NB learns from the statistics of your training set. On test set, it just uses the already learned statistics so portion of classes should not impact.

I observed that negative reviews tend to contain longer texts on average. would this have an impact on the way my classifier tries to predict?

Again nice question. Yes it does but that is not necessarily that bad. Longer text means that the probability of having larger number for term frequencies are higher. This is a feature itself! (Just think again that you already found a difference between classes. Well ... you want NB to do the same right?!!) The only thing is that if the difference in lengths of comments is very huge, then it might affect the value of features in a way that you need normalization (e.g. instead of counting terms as features, you can use TF-IDF which is a bounded score)

Hope it helps. Good Luck!

$\endgroup$
1
  • $\begingroup$ Thank you for your very detailed response. You very much clarified parts of my doubts. Concerning the last bit, I had already considered exploiting TFIDF, which is actually already implemented, but not enabled as I have not tested it. $\endgroup$ Jan 19, 2018 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.