Getting uniform distribution over topics from gensim's LDA?

Question

I am trying to learn topics distribution for each document in a corpus.

I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100) and when I try to infer vectors for each document I am getting uniform distribution over them. This is highly unlikely since documents are of different topics.

The relevant code snippet is:

#input : scipy sparse term-doc matrix (no_terms * no_docs)

corpus = gensim.matutils.Sparse2Corpus(term_doc)

lda = gensim.models.LdaModel(corpus, 100)

vec_gen = lda[corpus]

vecs = [vec for vec in vec_gen]

Now for each vector in vecs I am getting same probability for each topic.

Can anyone point out where I am going wrong?

Answer 1

I solved this issue. There is a parameter for minimum probability in gensim's LDA which is set to 0.01 by default. So topics with prob. < 0.01 are pruned from output.

Once I set min. prob to a very low value the results had all topics and their corresponding probability.

Answer 2

Probably not useful for OP, but a uniform distribution over your topics could be from certain terms persisting across documents. This discussion is more along the lines of true topic distribution vs. model coherence, but has implications tangential to this question which may help model performance.

Ex.)

doc_1 = ['A','B','C','X'],

doc_2 = ['A','C','D','Y'],

doc_3 = ['A','D','F','Y'],

doc_4 = ['A','B','F','X']

If it's the case that we would like to preserve a lower number of topics (maybe we're using LDA as an exploratory model rather than deploying it for regression or classification, or would like to use it to explain a subject matter to an audience).

We can see that 'A' persists across all documents, it could be justified that we could remove 'A' from the list of tokens for each document.

This is likely to be a case where we're analyzing documents that have to do with 'A' as a subject rather than a topic, or a more intuitive mind frame, suppose we're analyzing documents only about apples... Is it necessary to include apples if we already posit that the corpus is primarily composed of heart disease? Now, rather than burying the mathematical relationship we can see between 'B'&'X' and 'D'&'Y', we can decrease the number of topics required to accurately model the nuanced relationships.

Simply removing an over represented term can help expand the likelihood of very small (as a proportion) topics to be included, which, there's the argument of whether this can lead to a model which more closely approximates the true topic distribution within the subject, however, this approach assumes there are no terms that only share a relation through the subject term. An example of this would be if a new term 'Z' appeared in doc_2 and doc_4, the only exchangeable relationship is through 'A', so this could be lost if 'A' is removed. (Though it's very unlikely in a large corpus)