scikit-learn - Should I fit model with TF or TF-IDF?

Question

I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).

Looking at the example in the sklearn documentation, I was wondering why the LDA model is fit on a TF array, while the NMF model is fit on a TF-IDF array. Is there a precise reason for this choice?

Here is the example: http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-topics-extraction-with-nmf-lda-py

Also, any tips about how to find the best parameters (number of iterations, number of topics...) for fitting my models is well accepted.

Thank you in advance.

Answer 1

In the word sampling steps in LDA the word count is used as weights for the multinomial dist. Re-weighting the TF's by its IDF's would dispropotionally increase the chance of rare words being sampled, making them have a stronger influence in topic assignment. But rare words have equally low probabilities of showing up in all documents and topics, hence flattening out the topic distribution of each document.

That is a similar reason why it is recommended to remove stop-words before running LDA.

Answer 2

For topic modeling I have measured the within topic cosine distance and used that to optimize the number of topics derived. For each topic measure the pairwise cosine distance --> take the mean. Then for all topics, take the mean of the corresponding mean of the pairwise cosine distances between all vectors (within a topic). This will be your metric. Measure this value while iterating over varying numbers of topics and you should notice an elbow curve, telling you precisely where to cut the number of topics.

Answer 3

It is important to know that you can get the values of for the parameters you are seeking by using grid search and cross-validation. Take a look at the documentation on Scikit Learn for how to find the best parameters for fitting models and determine if it is helpful.