2
$\begingroup$

Reference: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html

I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning decay and learning offset, in this context and in general?

$\endgroup$
1
$\begingroup$

With batch you feed the entire data through each EM iteration. In the online implementation you feed only some of the data through each EM iteration (a "mini-batch"). From the sklearn user guide:

While the batch method updates variational variables after each full pass through the data, the online method updates variational variables from mini-batch data points.

From the paper about online LDA:

We then update $\lambda$ using a weighted average of its previous value and $\tilde\lambda$. The weight given to $\tilde\lambda$ is given by $\rho_{t} = (\tau_{0} + \tau)^{-\kappa}$, where $\kappa$ ∈ (0.5, 1] controls the rate at which old values of $\tilde\lambda$ are forgotten and $τ_{0} \ge 0$ slows down the early iterations of the algorithm.

In the sklearn implementation:

weight = np.power(self.learning_offset + self.n_batch_iter_, -self.learning_decay)

So learning_offset is $\tau_{0}$ which slows down early iterations, and learning_decay is $\kappa$ which controls rate at which old weights are forgotten.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @Wes. Why should the old weights be forgotten, and what do you mean when you say "slow down the early iterations"? $\endgroup$ – Minu Feb 15 '19 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.