Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
$\begingroup$
$\endgroup$
6
-
2$\begingroup$ Because LDA is based on term counts and document counts. $\endgroup$– Blue482Aug 6, 2017 at 15:16
-
$\begingroup$ @Blue482 many thanks for the answer :), may I ask you to explain more? I know the concept behind TFIDF and LDA, but I can't understand what will be wrong if we feed LDA with a vector which is times of terms counts and the weight in each document? $\endgroup$– sariiiAug 6, 2017 at 18:01
-
$\begingroup$ @Blue482 Also may I ask you to provide your answer in the second post,datascience.stackexchange.com/questions/21947/… because I can not comment in the first post as I explained. i made that as guest, and guest can not put comment, because of that I create my account and create another post, I really appreciate your help, needs your insight in the result. thanks :) $\endgroup$– sariiiAug 6, 2017 at 18:18
-
$\begingroup$ I've created another post in stackoverflow. as I can not follow your answer, may I ask you to follow there(I updated my answer there and it seems working but still some questions about the output)? as it would be hard managing in this way, I really appreciate your help in advance, stackoverflow.com/questions/45535277/… $\endgroup$– sariiiAug 6, 2017 at 19:18
-
$\begingroup$ @Blue482 I got why that is incorrect :|, it would be really good if you could add your explanations as answer, so I will accept that $\endgroup$– sariiiAug 7, 2017 at 4:55
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915
Direct quote:
In fact, Blei (who developed LDA), points out in the introduction of the paper of 2003 (entitled "Latent Dirichlet Allocation") that LDA addresses the shortcomings of the TF-IDF model and leaves this approach behind. LSA is compeltely algebraic and generally (but not necessarily) uses a TF-IDF matrix, while LDA is a probabilistic model that tries to estimate probability distributions for topics in documents and words in topics. The weighting of TF-IDF is not necessary for this.
That sums it up on the high level. It would be interesting to understand more technically, why the model would perform more poorly if TF-IDF is used. Actually, there is another reply in the SO link which claims that LDA can be improved with TF-IDF.
$\begingroup$
$\endgroup$
LDA is a word generating model, which assumes a word is generated from a multinomial distribution. It doesn't make sense to say 0.5 word(tf-idf weight) is generated from some distribution. In the Gensim implementation, it's possible to replace TF with TF-IDF, while in some other implementation, only integer input is allowed.
