I have set of newspaper articles. I want to identify important less frequent words in the set of newspaper articles. Currently I am using TF-IDF scores. However, it does not seem to be a good metric in my problem. Is there any better way of doing this?

Thank you in advance :)

  • $\begingroup$ Describe the problem. Non-stationarity? Neologisms? $\endgroup$ – Emre Nov 27 '17 at 16:16
  • $\begingroup$ @Emre This is a classification of newspaper atricles based on topics. For example sports, crime etc. Iwant to identify the important words to do this. $\endgroup$ – Volka Nov 28 '17 at 3:34
  • 1
    $\begingroup$ You want to classify or tag the article by topic? If so, you want a document classifier or topic model depending on whether the topics are fixed or not. How did you classifiy using the TF-IDF features and how well did it perform? I suppose you tried a bunch of classifiers and did not like the results? Try augmenting the features with the document's topic embedding. $\endgroup$ – Emre Nov 28 '17 at 4:00

You could try looking just at the IDF scores. Things like names/entities will score high, this may be desirable or undesirable depending on what you want.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.