I have set of newspaper articles. I want to identify important less frequent words in the set of newspaper articles. Currently I am using TF-IDF scores. However, it does not seem to be a good metric in my problem. Is there any better way of doing this?

Thank you in advance :)

  • $\begingroup$ Describe the problem. Non-stationarity? Neologisms? $\endgroup$
    – Emre
    Nov 27, 2017 at 16:16
  • $\begingroup$ @Emre This is a classification of newspaper atricles based on topics. For example sports, crime etc. Iwant to identify the important words to do this. $\endgroup$
    – Volka
    Nov 28, 2017 at 3:34
  • 1
    $\begingroup$ You want to classify or tag the article by topic? If so, you want a document classifier or topic model depending on whether the topics are fixed or not. How did you classifiy using the TF-IDF features and how well did it perform? I suppose you tried a bunch of classifiers and did not like the results? Try augmenting the features with the document's topic embedding. $\endgroup$
    – Emre
    Nov 28, 2017 at 4:00

1 Answer 1


You could try looking just at the IDF scores. Things like names/entities will score high, this may be desirable or undesirable depending on what you want.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.