I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my client. My biggest concern at this point is whether my models are reproducible or not. I understand that LDA models are probabilistic, so separate models run on the same data under the same parameters are not guaranteed to be identical.
However, for a model to be considered reliable you would expect duplicate models to come out looking roughly the same. A more reproducible model would be a more accurate representation of the text it is modeling. I wrote a script to find the average similarity between two distinct models (found here: https://github.com/HarryBaker/gensim/tree/topic2topic_seperate_models). The idea behind it is that I try to force a one to one relationship between two models' topics, and then find the average similarity of each match. The assumption is that a reproducible model would have a clear bijection between a duplicate.
From my early research it seems like training a model for longer increases the similarity of duplicate models. (Models trained under 500 iterations were more similar than those trained under 150 passes).
What I'm wondering is if there's been any papers or studies done on the reproducibility of LDA models, or if anyone has any ideas. I do not want to fix a seed. I don't want to be able to output an identical model--I want to be confident that for any model I create, a model trained under the same data and parameters will be almost the same.