While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. Is this an indication of an implementation mistake or is this normal and just an indication that I should use fewer topics?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It's normal: LDA tries to maximize the likelihood of the data according to the parameters by finding the right probabilities for the parameters. Usually at the beginning increasing the number of topics allows the model to separate topics more precisely and therefore obtain a higher likelihood. But at some point (depending on the data), increasing the number of topics cannot help the model anymore because the topics are already separated to the maximum and using all the topics would actually decrease the likelihood.
So it's a sign that you don't need that many topics. Note that it doesn't mean that the number of "used" topics is optimal for the application, it's often a balance to find.
