1
$\begingroup$

Let us say I have a LDA model trained on a corpus of text. I would like to know, for a newly given document, which one from the corpus is closet to it. But, to do so, I want to use probabilities provided by LDA.

Is it possible to measure a "similarity score" from probabilities?

I guess using classical formula of probability products would result in a very low value. I am newbie to NLP, I don't know much about literature.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

A LDA model provides for every document $d$ the posterior probabilities $p(z|d)$ for $d$ to belong to any topic $z$. These values form a distribution across topics $z_1,..,z_n$.

So obtaining which document is closer to a particular topic $z$ is easy: just sort the document by their probability for $z$.

To find the most similar document in a corpus $C$ to a doc $d$ according to the topics, assuming that all the $p(z|d)$ are available, requires a similarity measure between distributions. This can be done for instance with KL divergence, but there are other options.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Actually, I want to calculate similarity with probabilities provided by LDA (and avoid using KL or JS distances) Is it even possible ? $\endgroup$
    – aRedDish
    Jul 21, 2022 at 17:46
  • $\begingroup$ @aRedDish Yes, but this is what I tried to explain: with LDA you don't have one probability by document, you have a distribution of probs (a vector which sums to 1). So in order to compare 2 docs A and B with their LDA probs, you'd have to compare the full vector of probs fo A with the full vector of probs for B. And for this you need a way to measure similiarity between 2 distributions, I don't see any other way. However you could use other measures: cosine, bhattacharrya, etc. $\endgroup$
    – Erwan
    Jul 21, 2022 at 21:39
  • $\begingroup$ I got it. Let's say documents A & B share same dominant topic T (with prob of 0.9 for example, such that all other topic probs are close to 0). But LDA provides also probability (or "weight") of words (the keywords) for each topic so.. I wanted to use them through this process : assume B is given by words [w1, w2, ..,] the score I want is : score = prob(w1/A)*p(w2/A)*.. = (prob(w1/T)*prob(T/A))*(prob(w2/T)*prob(T/A)*... as you can see prob(T/A) is at power length(B) ! so the score I am having is very low ! from 1e-80 to 1e-5 (in the highest case) my question is : is this method ok ? $\endgroup$
    – aRedDish
    Jul 22, 2022 at 0:52
  • $\begingroup$ Well Imho the main problem is that you don't really need the LDA probs to do this: you can directly count the words in order to obtain score = prob(w1/A)*p(w2/A)*.. instead of involving (prob(w1/T)*prob(T/A))*(prob(w2/T)*prob(T/A)*... Also in theory you would have to take into account all the topics, so this would be a quite complex. Finally I don't see how the similarity between A and B is calculated here? $\endgroup$
    – Erwan
    Jul 22, 2022 at 9:21
  • $\begingroup$ Also, it's worth noting that the idea of computing similarity between documents through LDA probs is actually not very far from the principle of word embeddings: in both cases there is a representation in lower dimension based on co-occurrences. $\endgroup$
    – Erwan
    Jul 22, 2022 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.