Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Question

Reference

I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning decay and learning offset, in this context and in general?

Answer 1

With batch you feed the entire data through each EM iteration. In the online implementation you feed only some of the data through each EM iteration (a "mini-batch"). From the sklearn user guide:

While the batch method updates variational variables after each full pass through the data, the online method updates variational variables from mini-batch data points.

From the paper about online LDA:

We then update $\lambda$ using a weighted average of its previous value and $\tilde\lambda$. The weight given to $\tilde\lambda$ is given by $\rho_{t} = (\tau_{0} + \tau)^{-\kappa}$, where $\kappa$ ∈ (0.5, 1] controls the rate at which old values of $\tilde\lambda$ are forgotten and $τ_{0} \ge 0$ slows down the early iterations of the algorithm.

In the sklearn implementation:

weight = np.power(self.learning_offset + self.n_batch_iter_, -self.learning_decay)

So learning_offset is $\tau_{0}$ which slows down early iterations, and learning_decay is $\kappa$ which controls rate at which old weights are forgotten.