4
$\begingroup$

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?

$\endgroup$
  • 1
    $\begingroup$ Because LDA is based on term counts and document counts. $\endgroup$ – Blue482 Aug 6 '17 at 15:16
  • $\begingroup$ @Blue482 many thanks for the answer :), may I ask you to explain more? I know the concept behind TFIDF and LDA, but I can't understand what will be wrong if we feed LDA with a vector which is times of terms counts and the weight in each document? $\endgroup$ – sariii Aug 6 '17 at 18:01
  • $\begingroup$ @Blue482 Also may I ask you to provide your answer in the second post,datascience.stackexchange.com/questions/21947/… because I can not comment in the first post as I explained. i made that as guest, and guest can not put comment, because of that I create my account and create another post, I really appreciate your help, needs your insight in the result. thanks :) $\endgroup$ – sariii Aug 6 '17 at 18:18
  • $\begingroup$ I've created another post in stackoverflow. as I can not follow your answer, may I ask you to follow there(I updated my answer there and it seems working but still some questions about the output)? as it would be hard managing in this way, I really appreciate your help in advance, stackoverflow.com/questions/45535277/… $\endgroup$ – sariii Aug 6 '17 at 19:18
  • $\begingroup$ @Blue482 I got why that is incorrect :|, it would be really good if you could add your explanations as answer, so I will accept that $\endgroup$ – sariii Aug 7 '17 at 4:55
1
$\begingroup$

Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915

Direct quote:

In fact, Blei (who developed LDA), points out in the introduction of the paper of 2003 (entitled "Latent Dirichlet Allocation") that LDA addresses the shortcomings of the TF-IDF model and leaves this approach behind. LSA is compeltely algebraic and generally (but not necessarily) uses a TF-IDF matrix, while LDA is a probabilistic model that tries to estimate probability distributions for topics in documents and words in topics. The weighting of TF-IDF is not necessary for this.

That sums it up on the high level. It would be interesting to understand more technically, why the model would perform more poorly if TF-IDF is used. Actually, there is another reply in the SO link which claims that LDA can be improved with TF-IDF.

$\endgroup$
0
$\begingroup$

LDA is a word generating model, which assumes a word is generated from a multinomial distribution. It doesn't make sense to say 0.5 word(tf-idf weight) is generated from some distribution. In the Gensim implementation, it's possible to replace TF with TF-IDF, while in some other implementation, only integer input is allowed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.