3
$\begingroup$

I'm training a corpus consisting of 200000 reviews into positive and negative reviews using a Naive Bayes model, and I noticed that performing TF-IDF actually reduced the accuracy (while testing on test set of 50000 reviews) by about 2%. So I was wondering if TF-IDF has any underlying assumptions on the data or model that it works with, i.e. any cases where accuracy is reduced by the use of it?

$\endgroup$
  • $\begingroup$ Maybe the dataset you contain is vary varied and the frequencies of the words don't matter to come to the conclusion? $\endgroup$ – Hima Varsha Aug 25 '16 at 13:32
  • $\begingroup$ But if that was the case, shouldn't it give a performance on par with the non-use of it? If the frequencies don't account for the conclusion, shouldn't TF-IDF do zilch to the accuracy (as we assume it doesn't really matter either way)? $\endgroup$ – Train Heartnet Aug 25 '16 at 13:43
  • $\begingroup$ So when you say you don't use tf-idf, what are your features then? Just a simple BOG? $\endgroup$ – Hima Varsha Aug 25 '16 at 17:03
  • $\begingroup$ My features are unigrams and bigrams formed from the review texts (including stopwords) without any feature selection other than excluding features occurring only once. I'm not sure what "BOG" stands for. $\endgroup$ – Train Heartnet Aug 25 '16 at 18:09
  • $\begingroup$ When applying TF-IDF, does your output vocabulary has the same exact dimensions as your input vocabulary? $\endgroup$ – Ricardo Cruz Aug 26 '16 at 9:52
4
$\begingroup$

The IDF part of TF-IDF gives less weight to a word if it occurs in a large fraction of the documents in your corpus. However, this doesn't necessarily mean that the word is unimportant for distinguishing your two classes. A word which is common in your corpus, but which also occurs substantially more often in one class than the other, could very well be quite valuable in distinguishing the classes. This can especially be true if your set is not balanced between the two classes.

For example: Suppose that 95% of negative reviews contained the term "boring", while only 5% of the positive reviews contained "boring". If this occurred in a balanced set of negative and positive reviews, that means that half the total reviews would contain "boring", and this might be one of the terms whose weight is more strongly suppressed by IDF, despite the fact that it's obviously a valuable term for distinguishing the classes. If instead 90% of the reviews are negative (with "boring" again occurring in 95% of negative reviews and 5% of positive ones), then a full 86% of your documents contained the term, and its weight might be highly suppressed by IDF despite its obvious importance.

In essence, the problem is that TF-IDF is agnostic to the class labels in your training set. One alternative to TF-IDF for text classification that doesn't have this flaw is Bi-Normal Separation (BNS), see for example this paper:

Forman, G. BNS Feature Scaling: An Improved Representation over TF-IDF for SVM Text Classification. Hewlett-Packard Labs Tech Report HPL-2007-32R1, 2007.(link to PDF)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.