1
$\begingroup$

I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus.

My understanding of TF-IDF is that it is calculated per token per document, so the calculated weights are relative to a given document in the corpus. Is there a way to use TF-IDF to recover the most significant terms in the entire corpus, or is this the wrong approach? If the latter, what would be a more appropriate NLP approach?

$\endgroup$
  • $\begingroup$ 'Important' is different for different use cases and 'Important' is different for different people. How you define 'Important' is different. If a word is present in all the documents and important to you, you should use countvectorizer, because tf-idf will lower the importance of the words if it is present in all the documents. $\endgroup$ – vbrises Jun 6 at 14:35
  • $\begingroup$ @Vishal That is a good point - there very well may be words that are "important" to me that are present in all or most of the documents. My use case is similar to how Yelp extracts the "highlights" from all of the reviews for a particular business. So it sounds like TF-IDF would not be appropriate here? $\endgroup$ – Julian Jun 6 at 14:54
  • $\begingroup$ You should compare output of different approaches like count-vectorizer and tf-idf vectorizer and see which model's output more suits to your need. $\endgroup$ – vbrises Jun 6 at 15:15
1
$\begingroup$

Provided limited information & context you have provided, I would suggest you to look for feature selection when each dimension belongs to a word. Feature selection will give you most important words. Most important words in the sense, words deciding the decision surface of the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.