1
$\begingroup$

What are the advantages/disadvantages of using tfidf on n-grams generated through countvectorizer when your end goal is to see the frequent occurring terms in the corpus with the occurrence percentage?

$\endgroup$
1
$\begingroup$

First, CountVectorizer produces a matrix of token counts, not TF-IDF weights. In order to obtain TF-IDF weights you would have to use TfidfVectorizer.

If the goal is to study term frequency, there is no point using TF-IDF since TF-IDF weights are different from frequency. TF-IDF is used to reduce the weight of tokens which appear frequently compared to tokens which appear rarely. Moreover, TF-IDF weights are at the level of a document, so they cannot be used as a measure of global comparison across all documents. Across documents you could use the IDF (Inverse Document Frequency) part only, but then why not simply use Document Frequency.

Note that the same applies to a token count matrix: the values are at the level of a document. In order to find the global frequency one has to sum across the documents for every token.

Finally if you are trying to find the most frequent terms/n-grams of any length, it's difficult to compare frequencies between n-grams of different length. Additionally you're going to find real "terms" mixed with frequent grammatical constructs, for example "it is" is not a term but it's a frequent n-gram.

$\endgroup$
3
  • $\begingroup$ so in term frequency case using tfidf on n-grams is just a redundant step? My thoughts were that it scales it. $\endgroup$
    – Swarley
    Jun 15 at 12:09
  • $\begingroup$ @Tangent TFIDF is not like scaling, it combines term frequency (TF) with Inverse Document Frequency (IDF). the IDF part is meant to increase the weight of rare tokens compared to frequent tokens, so it goes in the opposite direction of frequency. It's essentially a heuristic method meant to make frequent tokens (typically stop words) less important when comparing TFIDF vectors. In other words it's supposed to make vectors more representative of the semantics by increasing the importance of content words. $\endgroup$
    – Erwan
    Jun 15 at 13:19
  • 1
    $\begingroup$ Note that terminology extraction is not as simple as finding the most frequent "terms", it's a quite complex task. $\endgroup$
    – Erwan
    Jun 15 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.