NLTK Sklearn Genism Text to Topic

Question

I aint no data scientist/machine learner.

What Im Lookin for

text = "Donald Trump became the president of America"
#some data science works
print(topics) 
#prints ["politics"]

text = "Rihanna is starring in the new movie The Inception 2"
#some data science works
print(topics) 
#prints ["movie","music"]

What I can do

I can extract words like Donald Trump,America,Rihanna using POS

I can get huge paragraphs/lists of articles/words on politics,movies etc and save them in a text file.

What i can't do

Make meaningful topics out of those words like sports,politics,movies

What i want you do to

Point me in the right specific direction towards solving this problem aka enlighten me

Answer 1

I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore.

You have basically two possibilities:

1. Classification of the text (Supervised learning).
   Supervised means that you need first to externally apply labels (for example manually by humans) to examples of texts (labels could be "politics" or "show") and then use one of the classification algorithms.
   You have extracted words from the text, so you could use a "bag of words" approach for classifying.
   There exist adaptations of classification algorithms (multi-label classification) in order to provide multiple labels (such as one text is labelled both with "music" and "movie").
   You can find text corpora already pre-labelled, to train the algorithm and partly avoid the manual effort.

2. Clustering the text, topic modelling (unsupervised learning).
   In this case you do not need to provide examples with labels but the algorithm will cluster the texts according to parameters like the similarity of two texts or on keywords/topics extracted from the texts.
   Although this method do not require labeled examples, the clusters you get as output will require fine tuning, e.g. number of clusters to produce, names of the clusters, etc.

Since you mentioned SKlearn, you can find some directions on their web site: Text Feature Extraction.

Here is a comprehensive (bit older) summary: Machine Learning in Automated Text Categorization by Fabrizio Sebastiani

Answer 2

To build off Mashimo's answer, one straightforward approach for topic modeling is "Latent Dirichlet Allocation" (LDA). The basic idea behind LDA is explained in this really good tutorial. Essentially, documents are assumed to be composed of mixtures of topics, which are in turn composed of mixtures of words. If we knew the topic and document distributions, we could generate new documents using a probabilistic model. In LDA, we run this process in reverse to infer the topic and document distributions given the documents.

In the tutorial, the author uses LDA to find topics within Sarah Palin's emails, which is not too dissimilar to what you're trying to do, if I understand correctly. For example, one topic is composed of the words "gas, oil, pipeline, agia, project, natural, north", which corresponds roughly to the topic "energy" or "gas". Note that LDA doesn't name the topics for you; you'll have to apply your own judgment to construct a sensible name for the group of words comprising a topic.

LDA has been implemented in packages like Gensim. To see how to use LDA in Python, you might find this SpaCy tutorial (which covers a lot of stuff in addition to LDA) useful.