In scikit-learn's LDA implementation, how can I sort the topics by frequency over the entire corpus?

Question

I've used scikit-learn to perform LDA topic modeling, and I'd ultimately like to sort the topics by saliency/frequency over the entire corpus, but I'm unsure how to do as such.

I've used pyldavis though there appears to be no way of extracting the frequencies and order, which I need to create certain visualisations.

This is basically what I've done so far in terms of topic modeling.

# Vectorize text data
vectorizer = countVectorizer() 
tf = vectorizer.fit_transform(df)

# Fit LDA model
LDA = LatentDirichletAllocation(n_components = k)
LDA.fit(tf)

From what I can gather, the best solution might be to use the document-topic distribution generated from LDA.transform(x), perhaps by manipulating the probabilities, but I don't know how. Any suggestions?

# Create document-topic distribution
doc_topics = LDA.transform(tf)

Thank you!

Answer 1

You should indeed use the distribution across topics by document $p(t|d)$. There are two options:

• The "classification" option: for every document select the topic which has the highest probability, i.e. label every document with the most likely topic. Then simply count the number of documents for every topic.
• The "probabilistic" option: for every topic $t$ sum the probability $p(t|d)$ across all the documents $d$. This means counting the proportion of each document which is considered as topic $t$.

Assuming there is a quite large number of documents, the two options usually produce similar results.