Questions tagged [lda]
Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.
126
questions
1
vote
0
answers
82
views
why does adding an LDA document vector with a word2vec word vector work well in LDA2vec?
In LDA the document weight vector represents the "weights" of each topic in the document. I think it's also valid to say, each row in the document vector corresponds to a word in the document, the ...
1
vote
1
answer
217
views
How to perform topic modelling on query search results
How can I model topics in the results returned by a search engine with higher weightage to documents ranked higher in the result set?
The use case that I am looking at involves extracting the most ...
1
vote
1
answer
769
views
What does online learning mean in Topic modeling (LDA) - Gensim
I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training."
So my assumption on what it means is - 'Once we have a ...
- 53
1
vote
2
answers
49
views
Reaching 100% accurray in Data Mining
I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results.
If I insepct the results of my Topic Model, the ...
- 151
2
votes
1
answer
41
views
Topic modelling on only 24 documents gives the same "topic" for any K
Description:
I have 24 documents, each one of around 2.5K tokens. They are public speeches.
My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
- 31
1
vote
0
answers
18
views
Searching for ressource recommendations (books,papers) to validate the results of a Topic Model (LDA)?
Hi I am building a Topic Model Process with Python.
To do this I am using the LDA.
However I am having trouble to determine the ideal amount of topics.
Currently I am using the CoherenceModel ...
- 151
0
votes
0
answers
84
views
To map topic to a document after topic modeling is done with LDA
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
- 11
2
votes
0
answers
89
views
Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?
I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
- 131
2
votes
1
answer
756
views
extract document topic vectors from lda model
how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
- 21
1
vote
0
answers
215
views
Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?
If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
- 131
8
votes
1
answer
1k
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
- 81
1
vote
1
answer
108
views
how to do topic modeling on very huge data?
When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
- 267
0
votes
1
answer
52
views
Which are the appropriate prameters for lda modeling?
I try to implement in R test for appropriate metrics for lda.
Here the way I try to use LDA
...
- 147
2
votes
1
answer
160
views
Where can I learn the complete mathematics involved in LDA?
I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems.
Where can I find good reading material which explains the ...
- 41
0
votes
2
answers
2k
views
LDA for sentiment analysis
As far as I understand it, LDA works by assuming that a corpus was written by a set of topics ands words corresponding to that topic by a specific distribution. I'm however not enterely sure what the ...
7
votes
4
answers
7k
views
BERT: it is possible to use it for topic modeling?
I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
- 193
0
votes
0
answers
70
views
Linking LDA topics to the input documents
I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
- 1
-1
votes
1
answer
52
views
How can I categoriese / classify a cluster of words?
I am just wondering if it is possible to classify word clusters?
For example if I provide you an array of words [bird,chicken,dock,park,apple,grapes,furits,juice]
...
- 345
0
votes
1
answer
59
views
LDA as a dimensionality reducer [closed]
I know how to use LDA as a classifier.
But how to use Linear Discriminant Analysis as a dimensionality reducer to reduce the number of features and apply logistic regression on top of it.
I am using R ...
2
votes
1
answer
2k
views
Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn
Reference
I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning?
Also, what is learning ...
- 795
0
votes
2
answers
700
views
Apply Labeled LDA on large data
I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
1
vote
2
answers
4k
views
How to Combine tfidf with LSTM in keras?
I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using ...
- 111
2
votes
0
answers
39
views
I have data of some movies and their subtitles.I want to classify them based on their ratings
I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
- 107
1
vote
0
answers
185
views
Model Joint Probability of N Words Appearing Together in a Sentence
Assume that we have a large corpus of texts to train with. Given N words as input, I want to model the joint probability $p(x_1, x_2, ..., x_N)$ of these words appearing together in a sentence. More ...
- 11
2
votes
1
answer
216
views
Predicting topics for customer reviews based on topics mapped to n-grams?
I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic.
Also I have a list of unigrams,bigrams and trigrams(not a part ...
2
votes
1
answer
2k
views
How to give names/labels to topics in LDA [duplicate]
I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused.
How can I use the information produced ...
- 129
2
votes
2
answers
422
views
What's beyond topic modeling?
I tried topic modeling (LDA, NMF) to extract insights from the data.
I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context?
(...
- 123
1
vote
1
answer
653
views
Industrial application(s) of LDA (latent Dirichlet allocation)?
LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining.
Question What are the industrial systems using LDA or may be some related models ? (May be Google/Amazon/ ... ? )
PS
I ...
1
vote
0
answers
800
views
Flexible Discriminant Analysis
I am studying the book "Elements of Statistical Learning". In chapter 12 it is given about the Generalized Linear Discriminant Analysis. In one of its section it is about Flexible Discriminant ...
- 131
1
vote
1
answer
790
views
How to compare the topic coherence between models of different number of topics?
If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence ...
- 11
6
votes
1
answer
555
views
How to split natural language script into segments?
I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
- 205
4
votes
1
answer
752
views
Why do we need the hyperparameters beta and alpha in LDA?
I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind:
First: Why do we need to add alpha and gamma every time we sample the equation ...
- 43
1
vote
1
answer
411
views
Implementation of LDA (Latent Dirichlet Allocation) for classification tasks
Until now I have used LDA only for topic modelling. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.
- 695
1
vote
1
answer
113
views
Cluster algorithm to group events in more general domains
I've a list of 1,300 news events, represented by only three terms coming from running LDA topic model on thousands of tweets. Here's some of them as an example:
...
- 33
0
votes
2
answers
358
views
Guided topic modeling: generating words from topics
I need to generate lists of words related to specific topics for a project. I am familiar with clustering methods of topic modeling such as LDA, but I have something else in mind. Are there any ...
3
votes
2
answers
2k
views
Sub topics with Latent Dirichlet Allocation
I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.
I've ...
- 131
5
votes
2
answers
986
views
Topic modeling for short length sentences
I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like ...
- 171
6
votes
1
answer
2k
views
Can I use euclidean distance for Latent Dirichlet Allocation document similarity?
I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
- 1,594
1
vote
0
answers
978
views
Perplexity increasing on Test DataSet in LDA (Topic Modelling)
I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created.
Unfortunately, perplexity is increasing with increased number of topics on ...
- 111
1
vote
1
answer
3k
views
How to map topic to a document after topic modeling is done with LDA
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ?
I am interested in clustering documents using unsupervised learning and ...
- 19
1
vote
1
answer
238
views
LDA vs Word2Vec vs Others for predicting recipients of a message
I'm investigating various NLP algorithms and tools to solve the following problem; NLP newbie here, so pardon my question if it's too basic.
Let's say, I have a messaging app where users can send ...
- 133
3
votes
1
answer
156
views
stable set PCA while adding features
Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular ...
- 198
1
vote
1
answer
511
views
Equally sized topics in Latent Dirichlet allocation
I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. ...
- 113
3
votes
0
answers
2k
views
why the accuracy of LDA model is always changing and also is high
Let’s explain the whole goal firstly, then go through the question.
I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents.
My dataset ...
- 331
2
votes
0
answers
1k
views
fitting classifier object of type 'int' has no len()
We have LDA topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics.
Also, we can evaluate the model we have created. one ...
- 331
2
votes
0
answers
9k
views
Found array with dim 3. Estimator expected <= 2
I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model.
I decided to use multinomial SVM as the evaluator.
...
- 331
17
votes
3
answers
24k
views
Why should we not feed LDA with TF-IDF input?
Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
- 171
2
votes
1
answer
3k
views
Apply SVM on LDA in python
hope someone kindly put time here,
my approach is like this:
TFIDF -> LDA -> SVM
I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document ...
- 31
-1
votes
1
answer
2k
views
Words from LDA output pyspark machine learning
I built a pipeline for an LDA model using pyspark's machine learning.
Here is my code:
...
1
vote
1
answer
672
views
News topic detection and categorization
If I want to get how many and what kind of topics are covered by New York Times each week from a bag of words model(All the news covered by NYT in a week) how should I approach? Using traditional ...
- 11