Questions tagged [lda]
Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.
126
questions
2
votes
1
answer
3k
views
How to compare LDA and TF-IDF?
I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked ...
- 121
4
votes
1
answer
5k
views
Difference between LDA and Naive Bayes
LDA: linear discriminant analysis
Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values.
Suppose our data ...
- 1,346
0
votes
1
answer
351
views
Quantifying the Reproducibility of LDA Models
I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my ...
- 11
2
votes
2
answers
149
views
Proceeding with various methods for news recommendation
I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation.
I went through this paper which mentions different methods used for news recommendation (at 7th ...
1
vote
1
answer
583
views
Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities
Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document.
I know that I have to compute the ...
- 11
1
vote
0
answers
272
views
Scikit Learn Latent Dirichlet Allocation overload my SWAP and/or RAM
1) After using LDA of scikit learn I realized it overloads the swap, but never reach more than 20% of the RAM. Do you have any idea why ? I can't really see if this is a problem, but still I would ...
- 1,307
4
votes
2
answers
1k
views
Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis
I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
- 41
0
votes
1
answer
2k
views
Goodness of fit metric to compare topic models NMF vs LDA
Does anyone have a good idea for how to compare topic modeling done by NMF and LDA? Let's say I fit LDA to a dataset and generate topic-word and document-topic distributions--I can use perplexity, for ...
- 118
1
vote
1
answer
121
views
Supervised Recommendation System trained on labeled phrase segments
I have a big collection of phrase segments (not whole ones) with user provided labels based on text similarity:
...
4
votes
3
answers
2k
views
scikit-learn - Should I fit model with TF or TF-IDF?
I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).
Looking at the example in ...
- 41
0
votes
1
answer
599
views
Classifier on top of LDA topic vectors?
I have training data in form of pair of documents with an associated label - {doc1, doc2, label}. Label is defined as function of pair of documents.
Now I want to build a model which can predict the ...
- 3,775
2
votes
2
answers
1k
views
Getting uniform distribution over topics from gensim's LDA?
I am trying to learn topics distribution for each document in a corpus.
I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100) and ...
- 3,775
2
votes
1
answer
316
views
How to get columns from unsorted rows in Pandas? (MALLET)
My data (the doc-topics output from a MALLET topic model) has the following shape:
...
- 21
2
votes
1
answer
2k
views
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
I can not seem to find it anywhere, even not from the author of LDA, ...
- 55
1
vote
1
answer
125
views
Is what I did supervised or unsupervised Machine learning?
My goal is to get a smartphone names from Twitter. So this is what I followed:
1- I extracted 100K tweets using the keyword “smartphone”.
2- I Applied LDA after applying ngram tokenization and ...
- 11
4
votes
1
answer
2k
views
How can I run Labeled LDA over one textual document?
I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help ...
- 41
1
vote
2
answers
535
views
Memory error - Hierarchical Dirichlet Process, HDP gensim
I am running Hierarchical Dirichlet Process, HDP using gensim in Python but as my corpus is too large it is throwing me following error:
...
- 111
2
votes
0
answers
132
views
Extending Author-Topic LDA
I've been trying to extend the LDA and wanted some help, direction and insight.
Can Author-Topic LDA be used as a document "category" model? The premise of the Author-Topic model is that multiple ...
- 21
2
votes
1
answer
455
views
In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?
In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices?
I understand that I will not get ...
- 23
2
votes
3
answers
2k
views
replicability / reproducibility in topic modeling (LDA)
If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) ...
- 45
5
votes
2
answers
396
views
Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?
I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics. I combined all of these documents into a LDA and got a topic like the one ...
- 821
5
votes
3
answers
2k
views
Need help with LDA for selecting features
I am currently selecting features of products by using LDA to group 6000 keywords of product into topics.
Here is the sample of my dataset after being organized into list of keywords for each product ...
- 303
9
votes
5
answers
11k
views
Tutorials on topic models and LDA
I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with ...
- 191
9
votes
3
answers
15k
views
Clustering of documents using the topics derived from Latent Dirichlet Allocation
I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(...
- 211
25
votes
2
answers
35k
views
What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?
LDA has two hyperparameters, tuning them changes the induced topics.
What does the alpha and beta hyperparameters contribute to LDA?
How does the topic change if one or the other hyperparameters ...
- 2,340
62
votes
6
answers
32k
views
Latent Dirichlet Allocation vs Hierarchical Dirichlet Process
Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
- 2,340