I've used scikit-learn to perform LDA topic modeling, and I'd ultimately like to sort the topics by saliency/frequency over the entire corpus, but I'm unsure how to do as such.
I've used pyldavis though there appears to be no way of extracting the frequencies and order, which I need to create certain visualisations.
This is basically what I've done so far in terms of topic modeling.
# Vectorize text data vectorizer = countVectorizer() tf = vectorizer.fit_transform(df)
# Fit LDA model LDA = LatentDirichletAllocation(n_components = k) LDA.fit(tf)
From what I can gather, the best solution might be to use the document-topic distribution generated from LDA.transform(x), perhaps by manipulating the probabilities, but I don't know how. Any suggestions?
# Create document-topic distribution doc_topics = LDA.transform(tf)