For my site I'm working on a chat recommender that would recommend chats to users. Each chat has a title and description and my corpus is composed of many of these title and description documents. I was curious about training an LDA2Vec model, but considering the fact that this is a very dynamic corpus that would be changing on a minute by minute basis, it's not doable. I was thinking of just doing standard LDA, because LDA being a probabilistic model, it doesn't require any training, at the cost of not leveraging local inter-word relationships. The other added benefit of LDA2Vec was that I could get accurate labeled topics. So I thought, what if I use standard LDA to generate the topics, but then I use a pre-trained word2vec model whether that be trained locally on my corpus or a global one, maybe there's a way to combine both.

The junk below draws heavily from the stuff in the lda2vec paper:


topic 0: SpaceX, Nasa, Asteroids, Rover

w_j = target word

w_i = pivot word

d_j = document vector or what I'm calling the topic in my scenario

c_j = context vector

c_j = w_j + d_j

d_j = [v_1, v_2 ... v_n]

This is the loss function from the paper

Loss Function

I can then iterate each word w_j for each pivot w_i such that the loss is minimal:

so first iteration pivot word is w_i = SpaceX and w_j iterates over [Nasa, Asteroids, Rover]

second iteration pivot word is w_i = Nasa and w_j iterates over [SpaceX, Asteroids, Rover]

once it's done I'll have a document vector d_j, I'll find the closest word in the embedding space to d_j and I presume that's a good label for the topic. Either I'm being extremely naive or this maybe works. Any input would be much appreciated, thanks in advance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.