Questions tagged [lda]

Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.

Filter by
Sorted by
Tagged with
58
votes
6answers
26k views

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP ...
22
votes
2answers
28k views

What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

LDA has two hyperparameters, tuning them changes the induced topics. What does the alpha and beta hyperparameters contribute to LDA? How does the topic change if one or the other hyperparameters ...
9
votes
5answers
11k views

Tutorials on topic models and LDA

I would like to know if you people have some good tutorials (fast and straightforward) about topic models and LDA, teaching intuitively how to set some parameters, what they mean and if possible, with ...
9
votes
2answers
13k views

Why we should not feed LDA with tfidf

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
9
votes
3answers
13k views

Clustering of documents using the topics derived from Latent Dirichlet Allocation

I want to use Latent Dirichlet Allocation for a project and I am using Python with the gensim library. After finding the topics I would like to cluster the documents using an algorithm such as k-means(...
7
votes
4answers
5k views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
6
votes
1answer
307 views

How to split natural language script into segments?

I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
5
votes
3answers
2k views

Need help with LDA for selecting features

I am currently selecting features of products by using LDA to group 6000 keywords of product into topics. Here is the sample of my dataset after being organized into list of keywords for each product ...
5
votes
2answers
357 views

Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?

I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics. I combined all of these documents into a LDA and got a topic like the one ...
5
votes
2answers
776 views

Topic modeling for short length sentences

I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like ...
5
votes
1answer
1k views

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
5
votes
0answers
503 views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using gensim library for topic modeling, more specifically LDA. I have created my corpus, my dictionary and my lda model, and with the help of pyLDAvis library I visualize the results. When I ...
4
votes
1answer
450 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
4
votes
1answer
2k views

How can I run Labeled LDA over one textual document?

I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help ...
4
votes
2answers
478 views

Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis

I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
3
votes
1answer
130 views

stable set PCA while adding features

Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular ...
3
votes
0answers
363 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
3
votes
0answers
1k views

why the accuracy of LDA model is always changing and also is high

Let’s explain the whole goal firstly, then go through the question. I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents. My dataset ...
3
votes
3answers
1k views

scikit-learn - Should I fit model with TF or TF-IDF?

I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python). Looking at the example in ...
2
votes
2answers
364 views

What's beyond topic modeling?

I tried topic modeling (LDA, NMF) to extract insights from the data. I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context? (...
2
votes
3answers
2k views

replicability / reproducibility in topic modeling (LDA)

If I'm not wrong, topic modeling (LDA) is not replicable, i.e. it gives different results in different runs. Where does this come from (where does this randomness come from and why is it necessary?) ...
2
votes
2answers
1k views

Sub topics with Latent Dirichlet Allocation

I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this. I've ...
2
votes
1answer
3k views

How to compare LDA and TF-IDF?

I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked ...
2
votes
2answers
133 views

Proceeding with various methods for news recommendation

I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation. I went through this paper which mentions different methods used for news recommendation (at 7th ...
2
votes
1answer
27 views

Tweet Classification into topics- What to do with data

Good evening, First of all, I want to apologize if the title is misleading. I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into ...
2
votes
1answer
135 views

Where can I learn the complete mathematics involved in LDA?

I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems. Where can I find good reading material which explains the ...
2
votes
1answer
944 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning ...
2
votes
1answer
180 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
2
votes
1answer
3k views

Apply SVM on LDA in python

hope someone kindly put time here, my approach is like this: TFIDF -> LDA -> SVM I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document ...
2
votes
1answer
3k views

Difference between LDA and Naive Bayes

LDA: linear discriminant analysis Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values. Suppose our data ...
2
votes
1answer
400 views

In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?

In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices? I understand that I will not get ...
2
votes
0answers
12 views

Keep retweets during topic-modelling [duplicate]

I got a dataset made out of tweets and I need to classify them into topics. For topic modelling with LDA I have cleaned out the dataset (removing stopwords, mentions, symbols, etc). Do I need to ...
2
votes
0answers
25 views

Dealing with high dimensionality datasets

I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
2
votes
2answers
35 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
2
votes
0answers
18 views

Topic modelling on only 24 documents gives the same “topic” for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
2
votes
0answers
48 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
2
votes
0answers
34 views

I have data of some movies and their subtitles.I want to classify them based on their ratings

I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
2
votes
1answer
2k views

How to give names/labels to topics in LDA [duplicate]

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
2
votes
0answers
1k views

fitting classifier object of type 'int' has no len()

We have LDA topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics. Also, we can evaluate the model we have created. one ...
2
votes
0answers
7k views

Found array with dim 3. Estimator expected <= 2

I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model. I decided to use multinomial SVM as the evaluator. ...
2
votes
1answer
264 views

How to get columns from unsorted rows in Pandas? (MALLET)

My data (the doc-topics output from a MALLET topic model) has the following shape: ...
2
votes
0answers
125 views

Extending Author-Topic LDA

I've been trying to extend the LDA and wanted some help, direction and insight. Can Author-Topic LDA be used as a document "category" model? The premise of the Author-Topic model is that multiple ...
1
vote
1answer
631 views

News topic detection and categorization

If I want to get how many and what kind of topics are covered by New York Times each week from a bag of words model(All the news covered by NYT in a week) how should I approach? Using traditional ...
1
vote
1answer
229 views

TF-IDF for Topic Modeling

Can TF-IDF be used a sole method for Topic Modeling ? (I know there are better methods like LDA , LSA etc) I just want to understand if TF-IDF alone can help us in Topic modeling . If yes , can ...
1
vote
2answers
41 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
1
vote
1answer
367 views

Equally sized topics in Latent Dirichlet allocation

I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. ...
1
vote
1answer
37 views

Are LDA clusters identical across different runs?

for a given corpus are the Latent Dirichlet Allocation clusters for it is unique in general? How about the gensim multi-process implementation of LDA? are there ...
1
vote
1answer
542 views

Industrial application(s) of LDA (latent Dirichlet allocation)?

LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining. Question What are the industrial systems using LDA or may be some related models ? (May be Google/Amazon/ ... ? ) PS I ...
1
vote
2answers
951 views

Getting uniform distribution over topics from gensim's LDA?

I am trying to learn topics distribution for each document in a corpus. I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100) and ...
1
vote
1answer
11 views

What does updated alpha mean in LDA model?

I'm trying to understand LDA model by reading through implementations of the algorithm. Many implementations update alpha during training iterations with codes like: ...