Quantifying the Reproducibility of LDA Models

Question

I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my client. My biggest concern at this point is whether my models are reproducible or not. I understand that LDA models are probabilistic, so separate models run on the same data under the same parameters are not guaranteed to be identical.

However, for a model to be considered reliable you would expect duplicate models to come out looking roughly the same. A more reproducible model would be a more accurate representation of the text it is modeling. I wrote a script to find the average similarity between two distinct models (found here: https://github.com/HarryBaker/gensim/tree/topic2topic_seperate_models). The idea behind it is that I try to force a one to one relationship between two models' topics, and then find the average similarity of each match. The assumption is that a reproducible model would have a clear bijection between a duplicate.

From my early research it seems like training a model for longer increases the similarity of duplicate models. (Models trained under 500 iterations were more similar than those trained under 150 passes).

What I'm wondering is if there's been any papers or studies done on the reproducibility of LDA models, or if anyone has any ideas. I do not want to fix a seed. I don't want to be able to output an identical model--I want to be confident that for any model I create, a model trained under the same data and parameters will be almost the same.

Answer 1

As you already noticed, Topic Models aren't reproducible due to the probabilistic algorithm. I did some topic models on a large corpus (10k documents, 30M words), and generally, between two different topic models about 10% of the topics were incomparable. My corpus is mainly English, but contains some Latin texts, and for a given number of topics I had between 1 and 3 topics consisting of Latin language words predominantly.

What to do now depends on what you want to learn from your Topic Model. When you want to "interpret" the Topic Model (using methods from the humanities), the best thing is to commit to one topic model (i.e., fixing the random seed).

When you want to study some derived measures (like topic diversity), make sure that the derived measures taken from different topic models are "stable", i.e., centred around one value with small statistical fluctuations.

You can also try to create an ensemble of topic models and try some techniques to get an "average topic model" out of it. I have no experience yet in this business and cannot suggest techniques here.