I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that library uses the WarpLDA sampling algorithm insted of Gibbs?

Direct quote from Griffiths and Steyvers 'Finding scientific topics' paper: "... we can approximate p(w|K) by taking the harmonic mean of a set of values of p(w|z, K) when z is sampled from the posterior p(z|w, K) [(Kass and Raftery, 1995, Section 4.3)]. Our Gibbs sampling algorithm provides such samples, and the value of p(w|z, K) can be computed..."

Excuse me for being this basic, but does WarpLDA sampling algorithm provide same type of samples so I can use the harmonic mean as an indicator for the topic number?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.