17
$\begingroup$

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?

$\endgroup$
6
  • 2
    $\begingroup$ Because LDA is based on term counts and document counts. $\endgroup$
    – Blue482
    Aug 6, 2017 at 15:16
  • $\begingroup$ @Blue482 many thanks for the answer :), may I ask you to explain more? I know the concept behind TFIDF and LDA, but I can't understand what will be wrong if we feed LDA with a vector which is times of terms counts and the weight in each document? $\endgroup$
    – sariii
    Aug 6, 2017 at 18:01
  • $\begingroup$ @Blue482 Also may I ask you to provide your answer in the second post,datascience.stackexchange.com/questions/21947/… because I can not comment in the first post as I explained. i made that as guest, and guest can not put comment, because of that I create my account and create another post, I really appreciate your help, needs your insight in the result. thanks :) $\endgroup$
    – sariii
    Aug 6, 2017 at 18:18
  • $\begingroup$ I've created another post in stackoverflow. as I can not follow your answer, may I ask you to follow there(I updated my answer there and it seems working but still some questions about the output)? as it would be hard managing in this way, I really appreciate your help in advance, stackoverflow.com/questions/45535277/… $\endgroup$
    – sariii
    Aug 6, 2017 at 19:18
  • $\begingroup$ @Blue482 I got why that is incorrect :|, it would be really good if you could add your explanations as answer, so I will accept that $\endgroup$
    – sariii
    Aug 7, 2017 at 4:55

3 Answers 3

7
$\begingroup$

Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915

Direct quote:

In fact, Blei (who developed LDA), points out in the introduction of the paper of 2003 (entitled "Latent Dirichlet Allocation") that LDA addresses the shortcomings of the TF-IDF model and leaves this approach behind. LSA is compeltely algebraic and generally (but not necessarily) uses a TF-IDF matrix, while LDA is a probabilistic model that tries to estimate probability distributions for topics in documents and words in topics. The weighting of TF-IDF is not necessary for this.

That sums it up on the high level. It would be interesting to understand more technically, why the model would perform more poorly if TF-IDF is used. Actually, there is another reply in the SO link which claims that LDA can be improved with TF-IDF.

$\endgroup$
2
$\begingroup$

Latent Dirichlet Allocation (LDA) is a popular topic modeling algorithm that is often used to discover underlying topics in a collection of documents.

LDA operates by assuming that each document in the collection is a mixture of a small number of topics and that each word in the document is attributable to one of those topics. In simple terms, LDA is a generative probabilistic model that assumes documents are mixtures of topics and topics are mixtures of words.


TF-IDF is a frequency-based representation that captures the importance of words in a document relative to a corpus. It measures how often a word appears in a document and compensates for how common it is across all documents


The use of Term Frequency-Inverse Document Frequency (TF-IDF) vectors as input to LDA is not conceptually wrong, it might not yield the best results in all cases, can lead to suboptimal results or might not fully capture the underlying topic structure, and there are some conceptual differences that should be considered.

  • Semantic vs. Frequency-based Representation:

    • TF-IDF doesn't directly capture semantic relationships between words.
    • LDA aims to uncover the latent topics within a corpus and assigns each word a probability of belonging to each topic. It's more focused on discovering the underlying themes or topics in a collection of documents.
  • Potential Loss of Information:

    • Feeding TF-IDF values directly into LDA might lose some semantic information. LDA relies on word co-occurrence patterns to infer topics. By providing TF-IDF values, you're effectively downscaling the importance of common words, which might lead to topics being dominated by rare words.
  • Optimal Use of LDA:

    • LDA works better when it's fed with the raw frequency counts of words in a corpus. It's designed to discover topics based on these frequencies. While TF-IDF could be used, it's not its primary intended use.
  • Dimensionality:

    • LDA works with a fixed number of topics as a user-defined parameter. TF-IDF vectors can be very high-dimensional, and directly using them as input might result in topics being spread thinly across dimensions, making them less coherent and interpretable.
  • Normalization:

    • LDA relies on probabilities and distributions for topic assignments. TF-IDF vectors might not be normalized in a way that's optimal for this probabilistic framework.
  • Loss of Word Order:

    • LDA is a generative model that assumes word order is not important within documents. It only considers the distribution of words. TF-IDF, on the other hand, captures word importance and frequency in the document but discards word order. This can lead to a mismatch between the assumptions of LDA and the input representation.
$\endgroup$
0
$\begingroup$

LDA is a word generating model, which assumes a word is generated from a multinomial distribution. It doesn't make sense to say 0.5 word(tf-idf weight) is generated from some distribution. In the Gensim implementation, it's possible to replace TF with TF-IDF, while in some other implementation, only integer input is allowed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.