1
$\begingroup$

I've used scikit-learn to perform LDA topic modeling, and I'd ultimately like to sort the topics by saliency/frequency over the entire corpus, but I'm unsure how to do as such.

I've used pyldavis though there appears to be no way of extracting the frequencies and order, which I need to create certain visualisations.

This is basically what I've done so far in terms of topic modeling.

# Vectorize text data
vectorizer = countVectorizer() 
tf = vectorizer.fit_transform(df)
# Fit LDA model
LDA = LatentDirichletAllocation(n_components = k)
LDA.fit(tf)

From what I can gather, the best solution might be to use the document-topic distribution generated from LDA.transform(x), perhaps by manipulating the probabilities, but I don't know how. Any suggestions?

# Create document-topic distribution
doc_topics = LDA.transform(tf)

Thank you!

$\endgroup$

1 Answer 1

0
$\begingroup$

You should indeed use the distribution across topics by document $p(t|d)$. There are two options:

  • The "classification" option: for every document select the topic which has the highest probability, i.e. label every document with the most likely topic. Then simply count the number of documents for every topic.
  • The "probabilistic" option: for every topic $t$ sum the probability $p(t|d)$ across all the documents $d$. This means counting the proportion of each document which is considered as topic $t$.

Assuming there is a quite large number of documents, the two options usually produce similar results.

$\endgroup$
1
  • $\begingroup$ Thank you! I've managed to classify each document with a dominant topic as per the first option. $\endgroup$
    – Basht0n
    Aug 1, 2021 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.