7
$\begingroup$

I am working on a multilabel text classification problem with 10 labels. The dataset is small, +- 7000 items and +-7500 labels in total. I am using python sci-kit learn and something strange came up in the results. As a baseline, I started out with using the countvectorizer and was actually planning on using the tfidf vectorizer which I thought would work better. But it doesn't.. with the countvectorizer I get a performance of a 0.1 higher f1score. (0.76 vs 0.65)

I cannot wrap my head around why this could be the case? There are 10 categories and one is called miscellaneous. Especially this one gets a much lower performance with tfidf.

Does anyone know when or why tfidf could perform worse than count? I need to formalize this for my thesis.

$\endgroup$
2
  • $\begingroup$ I am using 10 fold cross validation and the countvectorizer consistently performs better. Thank you for this answer! I will look into it. I remove stopwords, both the countvectorizer as the tfidf vectorizer improve a specific amount when I remove them. $\endgroup$
    – user21169
    Commented Jul 5, 2016 at 15:30
  • $\begingroup$ What was your classifier algorithm (Naive Bayes, Random Forest, SVM, ...)? $\endgroup$ Commented Apr 17, 2018 at 19:22

2 Answers 2

6
$\begingroup$

There are a few possibilities. First, there is some variability in performance. It could have been only by chance that countvectorizer performed better than tf-idf. Did you use cross validation (with how many folds)? Is the superior performance of the countvectorizer reliable? I would compare performance across folds to make sure countvectorizer consistently performs better.

Second, if you find that countvectorizer reliably outperforms tf-idf on your dataset, then I would dig deeper into the words that are driving this effect. It may be that common words (words which will appear in multiple documents) are helpful in distinguishing between classes. There is substantial research that shows that use of some function words (e.g. first person singular pronouns, “I”) change depending on someone’s psychological state. Function words like pronouns are very common and would be down weighted in tf-idf, but given equal weight to rare words in countvectorizer. I’m not suggesting that first person singular pronouns in particular are driving your results, but it’s worth looking at what words are driving the effect. I would examine which words are important in both types of models, countvectorizer and tf-idf, and then think about whether the words that are most important for the countvectorizer make sense in the context of your text documents and labels. Also, are you removing stop words? You could also see how the models perform with and without stop words, which would be another way to test whether frequent words are actually helping you to distinguish between classes.

$\endgroup$
0
$\begingroup$

I'm curious. Which Algorithm did you use. On sklearn it is said that Naive Bayes Algorithm perform on word occurrence. They add that MultinomialNB for instance MAY work with ratio figure like TDIDF feature but it seems its not guaranteed.

I'm currently working on a similar problem than yours and using OneVsRestClassfiier wrapping classifier like SGDClassifier or LinearSVC, MultinomialNB gives a predicition score of 0 with TFIDF features! much worse than the other classifier.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.