Skip to main content

Questions tagged [lda]

Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.

Filter by
Sorted by
Tagged with
62 votes
6 answers
33k views

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
alvas's user avatar
  • 2,410
25 votes
2 answers
36k views

What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

LDA has two hyperparameters, tuning them changes the induced topics. What does the alpha and beta hyperparameters contribute to LDA? How does the topic change if one or the other hyperparameters ...
alvas's user avatar
  • 2,410
17 votes
3 answers
25k views

Why should we not feed LDA with TF-IDF input?

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
sariii's user avatar
  • 171
10 votes
3 answers
15k views

Clustering of documents using the topics derived from Latent Dirichlet Allocation

I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(...
Swan87's user avatar
  • 221
9 votes
5 answers
11k views

Tutorials on topic models and LDA

I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with ...
pedrobisp's user avatar
  • 191
8 votes
1 answer
1k views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
Tasos Lytos's user avatar
7 votes
4 answers
8k views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
xcsob's user avatar
  • 193
6 votes
1 answer
684 views

How to split natural language script into segments?

I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
A.D.'s user avatar
  • 205
6 votes
1 answer
2k views

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
PyRsquared's user avatar
  • 1,614
5 votes
3 answers
2k views

Need help with LDA for selecting features

I am currently selecting features of products by using LDA to group 6000 keywords of product into topics. Here is the sample of my dataset after being organized into list of keywords for each product ...
sylvia's user avatar
  • 303
5 votes
2 answers
398 views

Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?

I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics. I combined all of these documents into a LDA and got a topic like the one ...
Matt's user avatar
  • 821
5 votes
1 answer
1k views

Calculating optimal number of topics for topic modeling (LDA)

am going to do topic modeling via LDA. I run my commands to see the optimal number of topics. The output was as follows: It is a bit different from any other plots that I have ever seen. Do you think ...
Tahereh Maghsoudi's user avatar
5 votes
2 answers
1k views

Topic modeling for short length sentences

I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like ...
Elimination's user avatar
5 votes
1 answer
2k views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
lefnire's user avatar
  • 151
4 votes
2 answers
1k views

Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis

I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
Afou Lazyboy's user avatar
4 votes
1 answer
811 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
Mr. Almars's user avatar
4 votes
1 answer
6k views

Difference between LDA and Naive Bayes

LDA: linear discriminant analysis Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values. Suppose our data ...
user1825567's user avatar
  • 1,396
4 votes
1 answer
2k views

How can I run Labeled LDA over one textual document?

I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help ...
Amir Ladhar's user avatar
4 votes
3 answers
2k views

scikit-learn - Should I fit model with TF or TF-IDF?

I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python). Looking at the example in ...
Luca P.'s user avatar
  • 41
3 votes
2 answers
2k views

Sub topics with Latent Dirichlet Allocation

I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this. I've ...
tmthyjames's user avatar
3 votes
1 answer
193 views

How to generate synthetic text for LDA?

I am wanting to play around with LDA topic modelling, namely looking at the effects of document length, topic number etc all have on accuracy (I know it has been done elsewhere, but no one seems to ...
user5067291's user avatar
3 votes
1 answer
159 views

stable set PCA while adding features

Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular ...
Tagar's user avatar
  • 198
3 votes
0 answers
2k views

why the accuracy of LDA model is always changing and also is high

Let’s explain the whole goal firstly, then go through the question. I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents. My dataset ...
Maria's user avatar
  • 331
2 votes
2 answers
429 views

What's beyond topic modeling?

I tried topic modeling (LDA, NMF) to extract insights from the data. I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context? (...
izdi's user avatar
  • 123
2 votes
3 answers
2k views

replicability / reproducibility in topic modeling (LDA)

If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) ...
PLL's user avatar
  • 45
2 votes
2 answers
895 views

Topic Modeling: LDA vs LSA vs ToPMine

I am new to Topic Modeling. Is it possible to implement ToPMine in Python? In a quick search, I can't seem to find any Python package with ToPMine. Is ToPMine better than LDA and LSA? I am aware ...
Peter's user avatar
  • 179
2 votes
1 answer
220 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
Arghya Ganguly's user avatar
2 votes
1 answer
4k views

How to compare LDA and TF-IDF?

I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked ...
P.Tail's user avatar
  • 121
2 votes
2 answers
152 views

Proceeding with various methods for news recommendation

I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation. I went through this paper which mentions different methods used for news recommendation (at 7th ...
nishantbhardwaj2002's user avatar
2 votes
2 answers
1k views

Getting uniform distribution over topics from gensim's LDA?

I am trying to learn topics distribution for each document in a corpus. I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100) and ...
SHASHANK GUPTA's user avatar
2 votes
1 answer
97 views

Tweet Classification into topics- What to do with data

Good evening, First of all, I want to apologize if the title is misleading. I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into ...
Placido Pellegriti's user avatar
2 votes
1 answer
817 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
Syrinebh's user avatar
2 votes
1 answer
163 views

Where can I learn the complete mathematics involved in LDA?

I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems. Where can I find good reading material which explains the ...
Harish Reddy's user avatar
2 votes
1 answer
2k views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning ...
Minu's user avatar
  • 805
2 votes
1 answer
3k views

Apply SVM on LDA in python

hope someone kindly put time here, my approach is like this: TFIDF -> LDA -> SVM I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document ...
sari's user avatar
  • 31
2 votes
1 answer
462 views

In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?

In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices? I understand that I will not get ...
bma's user avatar
  • 23
2 votes
0 answers
81 views

What hyperparameter values does the LDA mallet model use by default? Is it true that the formula to calculate alpha = 5.0/n(topics)?

I am trying to figure out the default $\alpha$ & $\eta$ values used by mallet LDA, but there is not a lot of information on this. I did find a couple of answers, with no proper references, saying ...
SAMUEL DURAIVEL's user avatar
2 votes
0 answers
49 views

Evaluate Topic Modelling on synthetic data

I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
seaslug95's user avatar
2 votes
0 answers
13 views

Keep retweets during topic-modelling [duplicate]

I got a dataset made out of tweets and I need to classify them into topics. For topic modelling with LDA I have cleaned out the dataset (removing stopwords, mentions, symbols, etc). Do I need to ...
Placido Pellegriti's user avatar
2 votes
0 answers
87 views

What is the objective function that Latent Dirichlet Allocation (LDA) minimizes? [closed]

Latent Dirichlet Allocation (LDA) is a generative model which produces a list of topics. Each topic is represented by a distribution over words. Questions: What is the objective function that it ...
MAC's user avatar
  • 277
2 votes
1 answer
52 views

Dealing with high dimensionality datasets

I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
user75252's user avatar
2 votes
3 answers
248 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
Shivam Agrawal's user avatar
2 votes
1 answer
46 views

Topic modelling on only 24 documents gives the same "topic" for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
Luisda's user avatar
  • 31
2 votes
0 answers
97 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
MelaniaCB's user avatar
  • 131
2 votes
0 answers
40 views

I have data of some movies and their subtitles.I want to classify them based on their ratings

I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
Harshita Vemula's user avatar
2 votes
1 answer
2k views

How to give names/labels to topics in LDA [duplicate]

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
user3778289's user avatar
2 votes
0 answers
1k views

fitting classifier object of type 'int' has no len()

We have LDA topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics. Also, we can evaluate the model we have created. one ...
Maria's user avatar
  • 331
2 votes
0 answers
10k views

Found array with dim 3. Estimator expected <= 2

I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model. I decided to use multinomial SVM as the evaluator. ...
Maria's user avatar
  • 331
2 votes
1 answer
325 views

How to get columns from unsorted rows in Pandas? (MALLET)

My data (the doc-topics output from a MALLET topic model) has the following shape: ...
b4c3's user avatar
  • 21
2 votes
1 answer
2k views

How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?

How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation? I can not seem to find it anywhere, even not from the author of LDA, ...
blpasd's user avatar
  • 155