Skip to main content
Share Your Experience: Take the 2024 Developer Survey
24 votes

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

I wanted to point out, since this is one of the top Google hits for this topic, that Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Processes (HDP), and hierarchical Latent Dirichlet ...
user38663's user avatar
  • 241
7 votes

Why should we not feed LDA with TF-IDF input?

Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915 Direct quote: In fact, Blei ...
Lazer's user avatar
  • 171
5 votes
Accepted

Calculating optimal number of topics for topic modeling (LDA)

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing ...
prashant0598's user avatar
  • 1,501
4 votes

News topic detection and categorization

I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a gensim contributor, who works a lot with topic modelling. He has a couple of notebooks on his ...
WalternativE's user avatar
4 votes
Accepted

How to split natural language script into segments?

I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your ...
Emre's user avatar
  • 10.5k
4 votes
Accepted

What's beyond topic modeling?

You could use doc2vec to create vector representations of each document. Once you have all the vector representations you can use standard unsupervised clustering techniques like k-means, hierarchical ...
RossDeVito's user avatar
3 votes
Accepted

Equally sized topics in Latent Dirichlet allocation

It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization. There is a ...
Eugen's user avatar
  • 457
3 votes

Goodness of fit metric to compare topic models NMF vs LDA

NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document ...
NQD's user avatar
  • 221
3 votes

Tutorials on topic models and LDA

If you are looking for something simple to start with and easy to implement, I would recommend this. Beginners Guide to Topic Modeling
SVK's user avatar
  • 281
3 votes

Sub topics with Latent Dirichlet Allocation

I actually do not think your method is a good way to find subtopics. Consider a document X with a distribution of topics z. X is made up of a mixed model distribution of topic Z. If you just give a ...
Tophat's user avatar
  • 2,430
3 votes
Accepted

k-means and LDA for text classification: how to test accuracy?

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering. Secondly, you asked for testing ...
Kasra Manshaei's user avatar
3 votes
Accepted

TF-IDF for Topic Modeling

Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar. So technically it can indeed be done ...
Erwan's user avatar
  • 25.5k
2 votes

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Yee Whye Teh et al's 2005 paper Hierarchical Dirichlet Processes describes a nonparametric prior for grouped clustering problems. For example, the HDP helps in generalizing the Latent Dirichlet ...
Pluviophile's user avatar
  • 3,888
2 votes

Proceeding with various methods for news recommendation

Recommender Systems are a huge topic of its own right and goes without saying, with a lot of research going on. This book does a deep-dive into recommender systems and may not be something you want, ...
gokul_uf's user avatar
  • 141
2 votes

How to compare LDA and TF-IDF?

If you have the ground truth value of the documents (their topics) all you gotta do is pick a metric and compare results. For classification problems, as yours, a common metric would be f1_score; ...
epattaro's user avatar
  • 166
2 votes

Tutorials on topic models and LDA

I suggest trying Machine Learning Plu's Gensim tutorial. It will give you a holistic overview, on NLP and LDA, including: how to pre-process your data, do feature engineering and apply LDA.
Argyris's user avatar
  • 21
2 votes

Why should we not feed LDA with TF-IDF input?

Latent Dirichlet Allocation (LDA) is a popular topic modeling algorithm that is often used to discover underlying topics in a collection of documents. LDA operates by assuming that each document in ...
Pluviophile's user avatar
  • 3,888
2 votes
Accepted

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is ...
Thomas Cleberg's user avatar
2 votes

Topic modeling for short length sentences

A possible approach might be to use the most predictable and predictive word(s) in each cluster as its name(s). The following is inspired by Fisher's category utility used by the COBWEB algorithm. An ...
Lejafar's user avatar
  • 296
2 votes

Industrial application(s) of LDA (latent Dirichlet allocation)?

I can't answer for the systems in already existing companies, but I can definitely share an application of LDA in NLP. Latent Dirichlet Allocation is a popular technique use for topic modelling in ...
Kaustubh's user avatar
  • 146
2 votes

Predicting topics for customer reviews based on topics mapped to n-grams?

You might want to look at the Facebook starspace library - their examples are similar but with suggesting Facebook pages and hashtags. https://github.com/facebookresearch/StarSpace
garysieling's user avatar
2 votes

Reaching 100% accurray in Data Mining

In theory it's of course possible to reach perfect performance: if the algorithm can find what it needs in the features to correctly distinguish between classes (or clusters), then it will perform ...
Erwan's user avatar
  • 25.5k
2 votes

Reaching 100% accurray in Data Mining

I would like to make the argument that you actually cannot have statistically speaking 100.00% accuracy even in theory but you can get really close. However, you getting too close might mean that ...
pierround's user avatar
2 votes
Accepted

Are LDA clusters identical across different runs?

No, as there is randomness in the method implementation, for example here (in LdaModel of the gensim library). Hence, it can affect your final result in each run. Therefore, if you want to keep the ...
OmG's user avatar
  • 1,219
2 votes

How to generate synthetic text for LDA?

In general text is not generated artificially because this leads to non-realistic datasets. In the case of LDA it would be very easy to generate data using LDA itself since it's a generative model. ...
Erwan's user avatar
  • 25.5k
2 votes

Lost human names after 'Lemmatization' for topic modeling in python

Names qualify as a 'PROPN' postag. Adding it to the allowed_postags list should likely fix the issue. Edit: a simplified example ...
dx2-66's user avatar
  • 696
1 vote
Accepted

LDA vs Word2Vec vs Others for predicting recipients of a message

Why the detour with the topic? You can just learn a linear SVM to directly predict the recipient and avoid all the difficulties from topic modeling. What if a mail isn't about any of the previous ...
Has QUIT--Anony-Mousse's user avatar
1 vote

stable set PCA while adding features

If you're prepared to use a neural net, it might be worthwhile looking into spatial pyramid pooling. A spatial pyramid pooling layer can take inputs of varying sizes (eg from a convolutional layer) ...
Finn Reilly's user avatar
1 vote

Apply SVM on LDA in python

Following your code in your other thread: ...
Blue482's user avatar
  • 243

Only top scored, non community-wiki answers of a minimum length are eligible