24
votes
Latent Dirichlet Allocation vs Hierarchical Dirichlet Process
I wanted to point out, since this is one of the top Google hits for this topic, that Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Processes (HDP), and hierarchical Latent Dirichlet ...
- 241
7
votes
Why should we not feed LDA with TF-IDF input?
Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915
Direct quote:
In fact, Blei ...
- 171
5
votes
Clustering of documents using the topics derived from Latent Dirichlet Allocation
Assuming that LDA produced a list of topics and put a score against each topic for each document, you could represent the document and it's scores as a vector:
...
- 181
4
votes
News topic detection and categorization
I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a gensim contributor, who works a lot with topic modelling.
He has a couple of notebooks on his ...
- 141
4
votes
Accepted
How to split natural language script into segments?
I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your ...
- 10.5k
4
votes
Accepted
What's beyond topic modeling?
You could use doc2vec to create vector representations of each document. Once you have all the vector representations you can use standard unsupervised clustering techniques like k-means, hierarchical ...
- 145
4
votes
Accepted
Calculating optimal number of topics for topic modeling (LDA)
LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing ...
- 1,491
3
votes
Accepted
Equally sized topics in Latent Dirichlet allocation
It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization.
There is a ...
- 457
3
votes
Accepted
replicability / reproducibility in topic modeling (LDA)
LDA is Bayesian model. This means the desired result is a posterior probability distribution over the random vectors of interest (probability of topics etc. having seen some data).
Inference for ...
- 321
3
votes
Goodness of fit metric to compare topic models NMF vs LDA
NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document ...
- 211
3
votes
Tutorials on topic models and LDA
If you are looking for something simple to start with and easy to implement, I would recommend this.
Beginners Guide to Topic Modeling
- 281
3
votes
Sub topics with Latent Dirichlet Allocation
I actually do not think your method is a good way to find subtopics. Consider a document X with a distribution of topics z. X is made up of a mixed model distribution of topic Z. If you just give a ...
- 2,400
3
votes
Accepted
k-means and LDA for text classification: how to test accuracy?
First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering.
Secondly, you asked for testing ...
- 6,532
3
votes
Accepted
TF-IDF for Topic Modeling
Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar.
So technically it can indeed be done ...
- 25k
2
votes
Latent Dirichlet Allocation vs Hierarchical Dirichlet Process
Yee Whye Teh et al's 2005 paper Hierarchical Dirichlet Processes describes a nonparametric prior for grouped clustering problems. For example, the HDP helps in generalizing the Latent Dirichlet ...
- 3,608
2
votes
Proceeding with various methods for news recommendation
Recommender Systems are a huge topic of its own right and goes without saying, with a lot of research going on.
This book does a deep-dive into recommender systems and may not be something you want, ...
- 141
2
votes
How to compare LDA and TF-IDF?
If you have the ground truth value of the documents (their topics) all you gotta do is pick a metric and compare results. For classification problems, as yours, a common metric would be f1_score; ...
- 166
2
votes
Tutorials on topic models and LDA
I suggest trying Machine Learning Plu's Gensim tutorial. It will give you a holistic overview, on NLP and LDA, including: how to pre-process your data, do feature engineering and apply LDA.
- 21
2
votes
Accepted
Can I use euclidean distance for Latent Dirichlet Allocation document similarity?
Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is ...
- 1,515
2
votes
Topic modeling for short length sentences
A possible approach might be to use the most predictable and predictive word(s) in each cluster as its name(s). The following is inspired by Fisher's category utility used by the COBWEB algorithm.
An ...
- 296
2
votes
Industrial application(s) of LDA (latent Dirichlet allocation)?
I can't answer for the systems in already existing companies, but I can definitely share an application of LDA in NLP.
Latent Dirichlet Allocation is a popular technique use for topic modelling in ...
- 146
2
votes
Predicting topics for customer reviews based on topics mapped to n-grams?
You might want to look at the Facebook starspace library - their examples are similar but with suggesting Facebook pages and hashtags.
https://github.com/facebookresearch/StarSpace
- 154
2
votes
Reaching 100% accurray in Data Mining
In theory it's of course possible to reach perfect performance: if the algorithm can find what it needs in the features to correctly distinguish between classes (or clusters), then it will perform ...
- 25k
2
votes
Reaching 100% accurray in Data Mining
I would like to make the argument that you actually cannot have statistically speaking 100.00% accuracy even in theory but you can get really close. However, you getting too close might mean that ...
- 61
2
votes
Accepted
Are LDA clusters identical across different runs?
No, as there is randomness in the method implementation, for example here (in LdaModel of the gensim library). Hence, it can affect your final result in each run. Therefore, if you want to keep the ...
- 1,209
2
votes
How to generate synthetic text for LDA?
In general text is not generated artificially because this leads to non-realistic datasets. In the case of LDA it would be very easy to generate data using LDA itself since it's a generative model. ...
- 25k
2
votes
Lost human names after 'Lemmatization' for topic modeling in python
Names qualify as a 'PROPN' postag. Adding it to the allowed_postags list should likely fix the issue.
Edit: a simplified example ...
- 686
1
vote
Why should we not feed LDA with TF-IDF input?
Latent Dirichlet Allocation (LDA) is a popular topic modeling algorithm that is often used to discover underlying topics in a collection of documents.
LDA operates by assuming that each document in ...
- 3,608
1
vote
Only top scored, non community-wiki answers of a minimum length are eligible