Skip to main content
29 votes
Accepted

NLP - why is "not" a stop word?

Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible. It clearly makes sense to consider 'not' as a stop word ...
oW_'s user avatar
  • 6,377
24 votes

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

I wanted to point out, since this is one of the top Google hits for this topic, that Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Processes (HDP), and hierarchical Latent Dirichlet ...
user38663's user avatar
  • 241
13 votes
Accepted

Resume Parsing - extracting skills from resume using Machine Learning

I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely. Here are a few sources I found that ...
xhattam's user avatar
  • 146
10 votes

What is the difference between topic modeling and clustering?

The purpose of topic modeling methods is to discover the latent themes (topics) assumed to have generated the documents of a corpus. Topic modeling methods are built on the distributional hypothesis, ...
geompalik's user avatar
  • 411
7 votes

Why should we not feed LDA with TF-IDF input?

Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915 Direct quote: In fact, Blei ...
Lazer's user avatar
  • 171
7 votes
Accepted

Would Topic Modelling be classified as NLP or NLU?

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two? There is a confusion here: NLP is the whole domain of AI which deals with ...
Erwan's user avatar
  • 25.5k
6 votes
Accepted

Combine two sets of clusters

To compare two LDA topics, you're really trying to compute the distance between two probability distributions. One such measure that's commonly used in these circumstances is the Hellinger Distance. ...
Thomas Cleberg's user avatar
5 votes
Accepted

Calculating optimal number of topics for topic modeling (LDA)

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing ...
prashant0598's user avatar
  • 1,511
4 votes

NLTK Sklearn Genism Text to Topic

To build off Mashimo's answer, one straightforward approach for topic modeling is "Latent Dirichlet Allocation" (LDA). The basic idea behind LDA is explained in this really good tutorial. Essentially, ...
vbox's user avatar
  • 341
4 votes

NLTK Sklearn Genism Text to Topic

I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore. You have basically two possibilities: Classification of the text (...
Mashimo's user avatar
  • 41
4 votes

News topic detection and categorization

I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a gensim contributor, who works a lot with topic modelling. He has a couple of notebooks on his ...
WalternativE's user avatar
4 votes
Accepted

How to split natural language script into segments?

I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your ...
Emre's user avatar
  • 10.5k
4 votes
Accepted

What's beyond topic modeling?

You could use doc2vec to create vector representations of each document. Once you have all the vector representations you can use standard unsupervised clustering techniques like k-means, hierarchical ...
RossDeVito's user avatar
4 votes
Accepted

Do weights of keywords for each topic add up to 1 in topic modeling?

Yes, the weights would add up to 1. This is a dirchlet random variable - review documentation on scipy here - https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.dirichlet.html
Jayaram Iyer's user avatar
3 votes

Goodness of fit metric to compare topic models NMF vs LDA

NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document ...
NQD's user avatar
  • 221
3 votes
Accepted

Equally sized topics in Latent Dirichlet allocation

It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization. There is a ...
Eugen's user avatar
  • 457
3 votes

Tutorials on topic models and LDA

If you are looking for something simple to start with and easy to implement, I would recommend this. Beginners Guide to Topic Modeling
SVK's user avatar
  • 281
3 votes

Is it correct to create topic models using both train and test data?

You should only use your training set in this context though it may benefit you if you used an additional cross-validation set for feature selection. If you use training data in your feature selection ...
kevins_1's user avatar
  • 717
3 votes
Accepted

TF-IDF for Topic Modeling

Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar. So technically it can indeed be done ...
Erwan's user avatar
  • 25.5k
2 votes

Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

Yee Whye Teh et al's 2005 paper Hierarchical Dirichlet Processes describes a nonparametric prior for grouped clustering problems. For example, the HDP helps in generalizing the Latent Dirichlet ...
Pluviophile's user avatar
  • 3,928
2 votes

Tutorials on topic models and LDA

I suggest trying Machine Learning Plu's Gensim tutorial. It will give you a holistic overview, on NLP and LDA, including: how to pre-process your data, do feature engineering and apply LDA.
Argyris's user avatar
  • 21
2 votes

How can I discover topics in a social media data-set?

You can take a look at Latent Dirichlet Allocation. In my experience this does very well without too much effort. You need to remove words that don't help like stopwords (and in your case Twitter ...
Jan van der Vegt's user avatar
2 votes

Why should we not feed LDA with TF-IDF input?

Latent Dirichlet Allocation (LDA) is a popular topic modeling algorithm that is often used to discover underlying topics in a collection of documents. LDA operates by assuming that each document in ...
Pluviophile's user avatar
  • 3,928
2 votes

Compare two topic modelling sets

One method to compare the topics across two corpora and measuring their similarity is with Kullback-Leibler divergence, aka relative entropy. Kullback-Leibler divergence is a measure of how one ...
Brian Spiering's user avatar
2 votes
Accepted

Evaluating the result of topic modeling in a way that time matters

I think the issues comes from the fact that the item you are looking at (the word "sleeping") is a rare event, so the probability that you observe one is about 0. Technically, it is called a Poisson ...
AlainD's user avatar
  • 276
2 votes

Representation options of strings (keywords/topics) in models

Since (word-based) one-hot encoding and real-valued vector representations are already mentioned in the question, I would only add the n-gram representation, especially the character-based n-gram ...
leonard's user avatar
  • 126
2 votes

Reaching 100% accurray in Data Mining

In theory it's of course possible to reach perfect performance: if the algorithm can find what it needs in the features to correctly distinguish between classes (or clusters), then it will perform ...
Erwan's user avatar
  • 25.5k
2 votes

Reaching 100% accurray in Data Mining

I would like to make the argument that you actually cannot have statistically speaking 100.00% accuracy even in theory but you can get really close. However, you getting too close might mean that ...
pierround's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible