29
votes
Accepted
NLP - why is "not" a stop word?
Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible.
It clearly makes sense to consider 'not' as a stop word ...
oW_♦
- 6,264
24
votes
Latent Dirichlet Allocation vs Hierarchical Dirichlet Process
I wanted to point out, since this is one of the top Google hits for this topic, that Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Processes (HDP), and hierarchical Latent Dirichlet ...
- 241
13
votes
Accepted
Resume Parsing - extracting skills from resume using Machine Learning
I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely.
Here are a few sources I found that ...
- 146
10
votes
What is the difference between topic modeling and clustering?
The purpose of topic modeling methods is to discover the latent themes (topics) assumed to have generated the documents of a corpus. Topic modeling methods are built on the distributional hypothesis, ...
- 411
7
votes
How to give name to topics created using LDA?
I can suggest several papers on this topic:
Automatic Labelling of Topic Models
Automatic Labeling Hierarchical Topics
Representing Topics Labels for Exploring Digital Libraries
You can find more by ...
- 10.5k
7
votes
Why should we not feed LDA with TF-IDF input?
Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915
Direct quote:
In fact, Blei ...
- 171
7
votes
Accepted
Would Topic Modelling be classified as NLP or NLU?
Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two?
There is a confusion here: NLP is the whole domain of AI which deals with ...
- 25k
6
votes
Accepted
Combine two sets of clusters
To compare two LDA topics, you're really trying to compute the distance between two probability distributions.
One such measure that's commonly used in these circumstances is the Hellinger Distance. ...
- 1,515
4
votes
NLTK Sklearn Genism Text to Topic
To build off Mashimo's answer, one straightforward approach for topic modeling is "Latent Dirichlet Allocation" (LDA). The basic idea behind LDA is explained in this really good tutorial. Essentially, ...
- 341
4
votes
NLTK Sklearn Genism Text to Topic
I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore.
You have basically two possibilities:
Classification of the text (...
- 41
4
votes
News topic detection and categorization
I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a gensim contributor, who works a lot with topic modelling.
He has a couple of notebooks on his ...
- 141
4
votes
Accepted
How to split natural language script into segments?
I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your ...
- 10.5k
4
votes
Accepted
What's beyond topic modeling?
You could use doc2vec to create vector representations of each document. Once you have all the vector representations you can use standard unsupervised clustering techniques like k-means, hierarchical ...
- 145
4
votes
Accepted
Do weights of keywords for each topic add up to 1 in topic modeling?
Yes, the weights would add up to 1. This is a dirchlet random variable - review documentation on scipy here - https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.dirichlet.html
- 795
4
votes
Accepted
Calculating optimal number of topics for topic modeling (LDA)
LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing ...
- 1,491
3
votes
Goodness of fit metric to compare topic models NMF vs LDA
NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document ...
- 211
3
votes
Accepted
replicability / reproducibility in topic modeling (LDA)
LDA is Bayesian model. This means the desired result is a posterior probability distribution over the random vectors of interest (probability of topics etc. having seen some data).
Inference for ...
- 321
3
votes
Tutorials on topic models and LDA
If you are looking for something simple to start with and easy to implement, I would recommend this.
Beginners Guide to Topic Modeling
- 281
3
votes
Accepted
Equally sized topics in Latent Dirichlet allocation
It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization.
There is a ...
- 457
3
votes
Is it correct to create topic models using both train and test data?
You should only use your training set in this context though it may benefit you if you used an additional cross-validation set for feature selection. If you use training data in your feature selection ...
- 717
3
votes
Accepted
TF-IDF for Topic Modeling
Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar.
So technically it can indeed be done ...
- 25k
2
votes
Latent Dirichlet Allocation vs Hierarchical Dirichlet Process
Yee Whye Teh et al's 2005 paper Hierarchical Dirichlet Processes describes a nonparametric prior for grouped clustering problems. For example, the HDP helps in generalizing the Latent Dirichlet ...
- 3,608
2
votes
Tutorials on topic models and LDA
I suggest trying Machine Learning Plu's Gensim tutorial. It will give you a holistic overview, on NLP and LDA, including: how to pre-process your data, do feature engineering and apply LDA.
- 21
2
votes
How to discard trash topics from topic models?
I assume you have inspected your topic-by-word matrix, and that is why you say you have 'trash' topics. If no words are strongly associated with a given topic, it may not be a useful topic. If you ...
- 3,077
2
votes
How can I discover topics in a social media data-set?
You can take a look at Latent Dirichlet Allocation. In my experience this does very well without too much effort. You need to remove words that don't help like stopwords (and in your case Twitter ...
- 9,348
2
votes
How to give name to topics created using LDA?
If you don't want to dig into much NLP in that task, I suggest you to generate a set of most frequent NGrams (of lengths 2-5) from your documents and find the most distinct ngrams for each category ...
- 779
2
votes
Compare two topic modelling sets
One method to compare the topics across two corpora and measuring their similarity is with Kullback-Leibler divergence, aka relative entropy.
Kullback-Leibler divergence is a measure of how one ...
- 20.5k
2
votes
Accepted
Evaluating the result of topic modeling in a way that time matters
I think the issues comes from the fact that the item you are looking at (the word "sleeping") is a rare event, so the probability that you observe one is about 0. Technically, it is called a Poisson ...
- 256
Only top scored, non community-wiki answers of a minimum length are eligible