Latent Dirichlet Allocation (LDA) importance of document generation and Gibbs Sampling

Question

I am having trouble finding the correlation between the two seemingly uncorrelated parts of LDA.

What I understood from several videos is:

1. There is a document generation "part", which is the construction of two dirichlet distrubutions, one describing the distribution of documents to topics, and one describing the distirbution of topics to words.

2. There is the Gibbs sampling optimization "part", which takes in documents whose words have been assigned to topics, and at each iteration of the optimization the Gibbs sampling takes a step towards a somewhat monistic distribution of documents to topics and words to topics, i.e each document's word's belong more or less to the same topic, and each instance of a word in the corpus of documents belongs more or less to one topic.

My question is why is the first part needed? I assume its importance may be that the initial random assignment of topics need be from a dirichlet distribution, but I am not sure if it is the case.

Is there anything I am missing?

Answer 1

tldr: in LDA the "generation part" is used to describe the design of the model, not for practical applications.

LDA is a Bayesian generative model. The distinction between generative and discriminative models is addressed for example in this question or if you want more detail in this one.

As the name suggests, generative models can generate data. But in the case of LDA and other topic modelling models, it's never actually used for the task of generation because it would produce meaningless texts. In fact the generative aspect is more about the design of the model: the idea is to represent a model as a statistical generation process, i.e. by drawing some values out of a distribution represented by the parameters of the model. This is why these models are always described through a "generative story", which represents the design of the model. Note that this design involves a lot of assumptions and simplifications, this is why the practical ability to generate is not really useful in LDA.

In practice LDA is almost exclusively used to obtain a probabilistic clustering of a set of documents based on topics, with different topics represented as different distributions over the vocabulary. This is done by estimating the parameters of the model from the corpus, and this process is usually done with Gibbs sampling. Note that Gibbs sampling is a general estimation method, it's not specific to LDA.

In case this helps, I often explain the differences between the different tasks in computer science terms:

• For the generation task:
  • the input is a model parameters,
  • the output is a set of documents (potentially infinite).
• For the estimation task:
  • the input is a (finite) set of documents,
  • the output is the model parameters.
• For the "application" task:
  • the input is a model parameters and some input document $d$
  • the output is the posterior probabilities $p(t|d)$ for every topic $t$.