Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915
In fact, Blei (who developed LDA), points out in the introduction of the paper of 2003 (entitled "Latent Dirichlet Allocation") that LDA addresses the shortcomings of the TF-IDF model and leaves this approach behind. LSA is compeltely algebraic and generally (but not necessarily) uses a TF-IDF matrix, while LDA is a probabilistic model that tries to estimate probability distributions for topics in documents and words in topics. The weighting of TF-IDF is not necessary for this.
That sums it up on the high level. It would be interesting to understand more technically, why the model would perform more poorly if TF-IDF is used. Actually, there is another reply in the SO link which claims that LDA can be improved with TF-IDF.
Latent Dirichlet Allocation (LDA) is a popular topic modeling algorithm that is often used to discover underlying topics in a collection of documents.
LDA operates by assuming that each document in the collection is a mixture of a small number of topics and that each word in the document is attributable to one of those topics. In simple terms, LDA is a generative probabilistic model that assumes documents are mixtures of topics and topics are mixtures of words.
TF-IDF is a frequency-based representation that captures the importance of words in a document relative to a corpus. It measures how often a word appears in a document and compensates for how common it is across all documents
The use of Term Frequency-Inverse Document Frequency (TF-IDF) vectors as input to LDA is not conceptually wrong, it might not yield the best results in all cases, can lead to suboptimal results or might not fully capture the underlying topic structure, and there are some conceptual differences that should be considered.
Semantic vs. Frequency-based Representation:
- TF-IDF doesn't directly capture semantic relationships between words.
- LDA aims to uncover the latent topics within a corpus and assigns each word a probability of belonging to each topic. It's more focused on discovering the underlying themes or topics in a collection of documents.
Potential Loss of Information:
- Feeding TF-IDF values directly into LDA might lose some semantic information. LDA relies on word co-occurrence patterns to infer topics. By providing TF-IDF values, you're effectively downscaling the importance of common words, which might lead to topics being dominated by rare words.
Optimal Use of LDA:
- LDA works better when it's fed with the raw frequency counts of words in a corpus. It's designed to discover topics based on these frequencies. While TF-IDF could be used, it's not its primary intended use.
- LDA works with a fixed number of topics as a user-defined parameter. TF-IDF vectors can be very high-dimensional, and directly using them as input might result in topics being spread thinly across dimensions, making them less coherent and interpretable.
- LDA relies on probabilities and distributions for topic assignments. TF-IDF vectors might not be normalized in a way that's optimal for this probabilistic framework.
Loss of Word Order:
- LDA is a generative model that assumes word order is not important within documents. It only considers the distribution of words. TF-IDF, on the other hand, captures word importance and frequency in the document but discards word order. This can lead to a mismatch between the assumptions of LDA and the input representation.
LDA is a word generating model, which assumes a word is generated from a multinomial distribution. It doesn't make sense to say 0.5 word(tf-idf weight) is generated from some distribution. In the Gensim implementation, it's possible to replace TF with TF-IDF, while in some other implementation, only integer input is allowed.