Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I have categorized 800,000 documents into 500 categories using the Mahout topic modelling.
Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name for the group using any existing algorithm.
For the time being, I have used the following algorithm to arrive at the name for the topic:
For each topic
Take all the documents belonging to the topic (using the document-topic distribution output)
Run python nltk to get the noun phrases
Create the TF file from the output
name for the topic is the phrase (limited towards max 5 words)
Please suggest a approach to arrive at more relevant name for the topics.
If you don't want to dig into much NLP in that task, I suggest you to generate a set of most frequent NGrams (of lengths 2-5) from your documents and find the most distinct ngrams for each category using TF*IDF metric as sense importance of a particular ngram (normalizing measure by word count) and selecting those Ngrams that are used in a particular category and are not (or rarely) used in others.