I have a TF-IDF representation of labeled documents (say 5 labels).

Is there a way to estimate the top-n words that are most likely to contribute to the classification, before applying any model (such as NMF/LDA)? I presume it could be derived from the TF-IDF score in each label.

  • $\begingroup$ If you have labeled documents, why are you applying unsupervised models, non-negative matrix factorization (NMF) and latent Dirichlet allocation (LDA)? $\endgroup$ – Brian Spiering Sep 15 '17 at 12:42
  • $\begingroup$ I am confused about one thing. Why do you want to know the top-n words that will contribute the most to the classification? Is it to generate better features for your documents? Or to explain the classification better afterwards? $\endgroup$ – Valentin Calomme Sep 16 '17 at 12:43

The TF-IDF is a measure of the discriminating ability of a term in a document. The TF-IDF of a specific term increases if the term is more frequent in a specific document but decreases if it is frequent in the whole corpus.

The problem with TF-IDF is that while it is good in distinguishing the document from the corpus, it isn't good in distinguishing one label from another!

What you could try to compute is a variation of the TF-IDF, where in the nominator you would count the term frequency, not a specific document, but in the set of documents under the same label. The denominator would stay the same. This way, instead of getting the top terms for each document you could get the top terms for each label. This metric could give you a rough estimation of what you want.

Note that while the top term for each label would be the most important in classifying examples to the specific label, it isn't necessarily the best for classification in general. The best term for classification, theoretically, would be one that could divide your data in two equal parts.

|improve this answer|||||
  • $\begingroup$ What you mentioned is exactly what I'm looking for! $\endgroup$ – Adam Sep 17 '17 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.