Questions tagged [lda]
Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.
126
questions
62
votes
6
answers
32k
views
Latent Dirichlet Allocation vs Hierarchical Dirichlet Process
Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
- 2,340
25
votes
2
answers
35k
views
What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?
LDA has two hyperparameters, tuning them changes the induced topics.
What does the alpha and beta hyperparameters contribute to LDA?
How does the topic change if one or the other hyperparameters ...
- 2,340
17
votes
3
answers
24k
views
Why should we not feed LDA with TF-IDF input?
Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
- 171
9
votes
5
answers
11k
views
Tutorials on topic models and LDA
I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with ...
- 191
9
votes
3
answers
15k
views
Clustering of documents using the topics derived from Latent Dirichlet Allocation
I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(...
- 211
8
votes
1
answer
1k
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
- 81
7
votes
4
answers
7k
views
BERT: it is possible to use it for topic modeling?
I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
- 193
6
votes
1
answer
557
views
How to split natural language script into segments?
I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
- 205
6
votes
1
answer
2k
views
Can I use euclidean distance for Latent Dirichlet Allocation document similarity?
I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
- 1,604
5
votes
3
answers
2k
views
Need help with LDA for selecting features
I am currently selecting features of products by using LDA to group 6000 keywords of product into topics.
Here is the sample of my dataset after being organized into list of keywords for each product ...
- 303
5
votes
2
answers
396
views
Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?
I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics. I combined all of these documents into a LDA and got a topic like the one ...
- 821
5
votes
1
answer
993
views
Calculating optimal number of topics for topic modeling (LDA)
am going to do topic modeling via LDA. I run my commands to see the optimal number of topics. The output was as follows: It is a bit different from any other plots that I have ever seen. Do you think ...
5
votes
2
answers
986
views
Topic modeling for short length sentences
I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like ...
- 171
5
votes
1
answer
2k
views
How to choose threshold for gensim Phrases when generating bigrams?
I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA
...
- 151
4
votes
2
answers
1k
views
Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis
I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
- 41
4
votes
1
answer
752
views
Why do we need the hyperparameters beta and alpha in LDA?
I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind:
First: Why do we need to add alpha and gamma every time we sample the equation ...
- 43
4
votes
1
answer
5k
views
Difference between LDA and Naive Bayes
LDA: linear discriminant analysis
Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values.
Suppose our data ...
- 1,346
4
votes
1
answer
2k
views
How can I run Labeled LDA over one textual document?
I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help ...
- 41
4
votes
3
answers
2k
views
scikit-learn - Should I fit model with TF or TF-IDF?
I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).
Looking at the example in ...
- 41
3
votes
2
answers
2k
views
Sub topics with Latent Dirichlet Allocation
I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.
I've ...
- 131
3
votes
1
answer
163
views
How to generate synthetic text for LDA?
I am wanting to play around with LDA topic modelling, namely looking at the effects of document length, topic number etc all have on accuracy (I know it has been done elsewhere, but no one seems to ...
- 151
3
votes
1
answer
156
views
stable set PCA while adding features
Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular ...
- 198
3
votes
0
answers
2k
views
why the accuracy of LDA model is always changing and also is high
Let’s explain the whole goal firstly, then go through the question.
I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents.
My dataset ...
- 331
2
votes
2
answers
423
views
What's beyond topic modeling?
I tried topic modeling (LDA, NMF) to extract insights from the data.
I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context?
(...
- 123
2
votes
3
answers
2k
views
replicability / reproducibility in topic modeling (LDA)
If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) ...
- 45
2
votes
2
answers
746
views
Topic Modeling: LDA vs LSA vs ToPMine
I am new to Topic Modeling.
Is it possible to implement ToPMine in Python? In a quick search, I can't seem to find any Python package with ToPMine.
Is ToPMine better than LDA and LSA? I am aware ...
- 179
2
votes
1
answer
216
views
Predicting topics for customer reviews based on topics mapped to n-grams?
I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic.
Also I have a list of unigrams,bigrams and trigrams(not a part ...
2
votes
1
answer
3k
views
How to compare LDA and TF-IDF?
I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked ...
- 121
2
votes
2
answers
149
views
Proceeding with various methods for news recommendation
I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation.
I went through this paper which mentions different methods used for news recommendation (at 7th ...
2
votes
2
answers
1k
views
Getting uniform distribution over topics from gensim's LDA?
I am trying to learn topics distribution for each document in a corpus.
I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100) and ...
- 3,775
2
votes
1
answer
80
views
Tweet Classification into topics- What to do with data
Good evening,
First of all, I want to apologize if the title is misleading.
I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into ...
2
votes
1
answer
758
views
extract document topic vectors from lda model
how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
- 21
2
votes
1
answer
160
views
Where can I learn the complete mathematics involved in LDA?
I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems.
Where can I find good reading material which explains the ...
- 41
2
votes
1
answer
2k
views
Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn
Reference
I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning?
Also, what is learning ...
- 795
2
votes
1
answer
3k
views
Apply SVM on LDA in python
hope someone kindly put time here,
my approach is like this:
TFIDF -> LDA -> SVM
I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document ...
- 31
2
votes
1
answer
455
views
In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?
In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices?
I understand that I will not get ...
- 23
2
votes
0
answers
78
views
What hyperparameter values does the LDA mallet model use by default? Is it true that the formula to calculate alpha = 5.0/n(topics)?
I am trying to figure out the default $\alpha$ & $\eta$ values used by mallet LDA, but there is not a lot of information on this. I did find a couple of answers, with no proper references, saying ...
2
votes
0
answers
41
views
Evaluate Topic Modelling on synthetic data
I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
- 21
2
votes
0
answers
13
views
Keep retweets during topic-modelling [duplicate]
I got a dataset made out of tweets and I need to classify them into topics. For topic modelling with LDA I have cleaned out the dataset (removing stopwords, mentions, symbols, etc). Do I need to ...
2
votes
0
answers
76
views
What is the objective function that Latent Dirichlet Allocation (LDA) minimizes? [closed]
Latent Dirichlet Allocation (LDA) is a generative model which produces a list of topics. Each topic is represented by a distribution over words.
Questions:
What is the objective function that it ...
- 277
2
votes
1
answer
45
views
Dealing with high dimensionality datasets
I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
- 41
2
votes
3
answers
229
views
How to identify text similarity based on training data?
I have a set of documents (1 to 11) for which the labeling is done.
Lets Assume:
...
- 129
2
votes
1
answer
41
views
Topic modelling on only 24 documents gives the same "topic" for any K
Description:
I have 24 documents, each one of around 2.5K tokens. They are public speeches.
My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
- 31
2
votes
0
answers
90
views
Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?
I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
- 131
2
votes
0
answers
39
views
I have data of some movies and their subtitles.I want to classify them based on their ratings
I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
- 107
2
votes
1
answer
2k
views
How to give names/labels to topics in LDA [duplicate]
I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused.
How can I use the information produced ...
- 129
2
votes
0
answers
1k
views
fitting classifier object of type 'int' has no len()
We have LDA topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics.
Also, we can evaluate the model we have created. one ...
- 331
2
votes
0
answers
9k
views
Found array with dim 3. Estimator expected <= 2
I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model.
I decided to use multinomial SVM as the evaluator.
...
- 331
2
votes
1
answer
316
views
How to get columns from unsorted rows in Pandas? (MALLET)
My data (the doc-topics output from a MALLET topic model) has the following shape:
...
- 21
2
votes
1
answer
2k
views
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
I can not seem to find it anywhere, even not from the author of LDA, ...
- 55