# replicability / reproducibility in topic modeling (LDA)

If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) and what can be done to solve this issue or gain more stability across runs?
Thanks for the help.

LDA is Bayesian model. This means the desired result is a posterior probability distribution over the random vectors of interest (probability of topics etc. having seen some data).

Inference for many Bayesian models is done by Markov Chain Monte Carlo. Indeed the wiki on LDA suggests that Gibbs sampling is a popular inference technique for LDA. MCMC draws random samples to provide an approximation of the posterior distribution.

Variational inference methods should typically be deterministic, but I'm not too familiar with the VB inference for this particular model.

Also one can typically replicate runs of random algorithms by setting of random number generation seeds (if your purpose is scientific).

In either case if the results from a Bayesian model show huge variability in the the parameters of interest, it may be telling you that the model is not a good fit for the dataset, or the dataset is not big enough for the model you are fitting.

EDIT: I don't know which inference method (Gibbs, VB etc.) the backend of your software is using, so it's not possible to determine what type (if any) of randomisation is going on.

For scientific purposes, you'll probably want to read up some more on Bayesian inference. Standard software (e.g. the LDA in scikits.learn) will give you a summary of the outputs of the inference (e.g. most coders just want the best assignment of docs to topics). There's more information hanging around behind the scenes which you might be able to get access to, and could be useful.

E.g. (roughly) for scientific applications of Gibbs sampling methods we'd typically run multiple chains, drop the first N samples generated by each, and check if the resulting samples look like they came from the same distribution. If you're concerned about dependence on seeds etc. and your backend is the Gibbs sampler you will want to check out MCMC convergence diagnostics for this model.

• So, just that I understand well : 1) It's random because of Gibbs sampling. 2) By fixing the seed I can reproduce my results (yes purpose is scientific) 3) While I can reproduce my results the scientific issue is that there were no particular reason to favor this seed over another and thus for my results to be reliable they should not show too much variability across various seeds. – PLL Sep 25 '15 at 10:08

When you are using mallet, you can fix a random seed using the command line flag --random-seed. This makes your results reproducible.

It does not and cannot remove the fact, that different random seeds produce different topic models.