1
$\begingroup$

I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. However, what I find really weird is that they are all roughly the same size! Each topic catches around 5% of tokens and 5% of texts. In terms of the tokens, the smallest topic is 4.5% the largest 5.5%.

Can anybody suggest if this a 'normal' behaviour? This is the code I'm using:

ldafitted <- LDA(sentences.tm, k = K, method = "Gibbs",
             control = list(alpha = 0.1, # default is 50/k which would be 2.5.  a lower alpha value places more weight on having each document composed of only a few dominant topics
                            delta = 0.1, # default 0.1 is suggested in Griffiths and Steyvers (2004).
                            estimate.beta = TRUE,
                            verbose = 50, # print every 50th draw to screen
                            seed = 5926696,
                            save = 0,    # can save model every xth iteration
                            iter = 5000, 
                            burnin = 500,
                            thin = 5000, #  every thin iteration is returned for iter iterations. Standard is same as iter
                            best = TRUE)) #only the best draw is returned

In short: My question is if there are circumstances under which it is reasonable that Latent Dirichlet allocation will cluster text in topics of equal size? Or is it something I should be worried if it happens?

$\endgroup$
3
$\begingroup$

It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization. There is a system which has particles (words) and the particles can be in different states (topics). States with lower energy have a higher probability of being occupied than states with higher energy. Or simply: Since LDA is a dimensity reduction method that just compresses a words x documents matrix into a topics x documents and a words x topics matrix, the most effective way to do it is by maximizing the entropy which is done if the clusters get equal size. enter image description here

Just noticed that you can simply derive the above assumption from the dirichlet distribution itself: Take a look at the density (PDF) function and ignore the normalizing factor at the front and only assume symmetric alpha parameter. This results in a product term of X1...XK, that product term gets it's maximum if all X1...XK's have equal size, since the sum X1+X2+...+XK = 1, it's the same case as entropy above.

$\endgroup$
  • $\begingroup$ Thanks a lot for this. The question was actually still on my mind, even though I asked it some time ago. Can you provide an article or book mentioning this? What worries me a bit is that I have seen several applications where equally sized topics were not an issue. From your explanation, it seems like topics would almost always have roughly the same size... $\endgroup$ – JBGruber Jun 15 '18 at 19:25
  • 1
    $\begingroup$ Here you can find the physical explanation and the maximum entropy assumption: en.wikipedia.org/wiki/Canonical_ensemble The topics at least strive to have equal size. Maybe you can find more on this if you look at it from the matrix decomposition perspective: en.wikipedia.org/wiki/Matrix_decomposition QR decomposition looks promising. $\endgroup$ – Eugen Jun 19 '18 at 6:05
  • 1
    $\begingroup$ This looks even more promising: en.wikipedia.org/wiki/Non-negative_matrix_factorization $\endgroup$ – Eugen Jun 19 '18 at 6:11
  • $\begingroup$ Just noticed that you can simply derive the above assumption from the dirichlet distribution itself:[Link] wikimedia.org/api/rest_v1/media/math/render/svg/… Take a look at the density (PDF) function and ignore the normalizing factor at the front and only assume symmetric alpha parameter. This results in a product term of X1...XK, that product term gets it's maximum if all X1...XK's have equal size, since the sum X1+X2+...+XK = 1, it's the same case as entropy above. $\endgroup$ – Eugen Jun 27 '18 at 12:56
  • $\begingroup$ Small addition: LDA uses the en.wikipedia.org/wiki/Dirichlet-multinomial_distribution so you have to derive the above statement from it. $\endgroup$ – Eugen Jun 27 '18 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.