Questions tagged [lda]
Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.
126
questions
1
vote
1
answer
3k
views
How to map topic to a document after topic modeling is done with LDA
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ?
I am interested in clustering documents using unsupervised learning and ...
CommunityBot
- 1
1
vote
2
answers
4k
views
How to Combine tfidf with LSTM in keras?
I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using ...
- 81
8
votes
1
answer
1k
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
CommunityBot
- 1
2
votes
3
answers
229
views
How to identify text similarity based on training data?
I have a set of documents (1 to 11) for which the labeling is done.
Lets Assume:
...
CommunityBot
- 1
1
vote
1
answer
122
views
Topic alignment / topic modelling
What is the most efficient method for detecting whether the article is mostly about a specific topic, but without lots of data for training? My task is to determine how much a document is e.g. about ...
CommunityBot
- 1
4
votes
2
answers
1k
views
Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis
I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn ...
CommunityBot
- 1
5
votes
1
answer
2k
views
How to choose threshold for gensim Phrases when generating bigrams?
I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA
...
CommunityBot
- 1
1
vote
1
answer
86
views
Choice of the number of topics (clusters) in textual data
I have a social science background and I'm doing a text mining project.
I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm ...
CommunityBot
- 1
0
votes
2
answers
702
views
Apply Labeled LDA on large data
I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
CommunityBot
- 1
0
votes
1
answer
354
views
Topic modelling with many synonyms - how to extract 'latent themes'
Here's my corpus
...
CommunityBot
- 1
2
votes
1
answer
2k
views
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
I can not seem to find it anywhere, even not from the author of LDA, ...
CommunityBot
- 1
1
vote
1
answer
411
views
Implementation of LDA (Latent Dirichlet Allocation) for classification tasks
Until now I have used LDA only for topic modelling. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.
CommunityBot
- 1
17
votes
3
answers
24k
views
Why should we not feed LDA with TF-IDF input?
Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
- 3,608
0
votes
0
answers
29
views
number of topics in LDA model (coherence)
I want to fit a LDA model using RStudio, but there are some trouble in the determination of topics number.
Perplexity and coherence are suggested to do this work, so first I use LDA and perplexity ...
- 1
0
votes
0
answers
6
views
Is colorring each document and word as monochromatic as possible the goal of LDA in specific or all topic models in general?
In the video Training Latent Dirichlet Allocation: Gibbs Sampling, the Goal section at 7:32, the video says that:
Goal: Color each word with blue, green, red
Each article is as monochromatic as ...
- 101
0
votes
0
answers
36
views
Doing LDA (NLP task): Does it make sense to use tf-idf vectors?
Most implementations that I've seen of LDA seem to use simple word counts when giving the document-term frequency matrix.
What would happen if we were to give the tf-idf matrix instead of a simple ...
1
vote
2
answers
535
views
Memory error - Hierarchical Dirichlet Process, HDP gensim
I am running Hierarchical Dirichlet Process, HDP using gensim in Python but as my corpus is too large it is throwing me following error:
...
CommunityBot
- 1
2
votes
2
answers
746
views
Topic Modeling: LDA vs LSA vs ToPMine
I am new to Topic Modeling.
Is it possible to implement ToPMine in Python? In a quick search, I can't seem to find any Python package with ToPMine.
Is ToPMine better than LDA and LSA? I am aware ...
- 3,608
0
votes
0
answers
20
views
How the embedding model (x-vectors) trained?
I read this paper: X-Vectors: Robust DNN Embeddings for Speaker Recognition which describes how PyAnnote embedding block works.
I'm not sure I understand how the X-Vector model was trained and tested:
...
- 363
0
votes
0
answers
21
views
LDA calculations manually
my question: has anyone ever done LDA calculations manually? I have difficulty in manual calculation. can someone help me to teach me for lda calculations manually.
- 1
0
votes
0
answers
20
views
Algorithm of lda2vec in NLP
I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
- 1
1
vote
0
answers
172
views
Short Text Topic Modelling in Python
I have a large dataset of short reviews and I would like to find the most recurring themes. For this reason, I got into topic modeling.
I am looking for some good tutorials and references for short ...
- 23.8k
0
votes
1
answer
16
views
method of allocating documents to pre-defined classifications
I'm looking for a method of allocating documents (30K and growing) to a set of some 200 categories.
The categories will be user defined and will grow over time.
As my data is unlabelled my thought ...
- 4,609
0
votes
0
answers
9
views
Get top documents per topic from mallet lda
I'm using java mallet's LDA engine to determine topics for a set of (some 30K) documents.
I've managed to train the model and serialize/deserialize it as needed.
The question is how do I find the top '...
- 101
2
votes
1
answer
45
views
Dealing with high dimensionality datasets
I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
- 20.5k
1
vote
2
answers
44
views
Topic Modeling - n-grams or 1,2,3,...n-grams?
Do people use n-grams or 1,2,3,...n-grams in both matrix factorisation and generative models in Topic Modeling?
I've been trying to understand the basics of Topic Modeling and came to know that there ...
- 20.5k
1
vote
0
answers
111
views
Trying to use term document matrix as input to orange 3
I have a CSV file that has tokens as columns and documents as rows, where the rest of the cells are ints that represent term frequency. I'm trying to use this as input into Orange 3, but Orange 3 ...
- 1,635
0
votes
1
answer
156
views
Lost human names after 'Lemmatization' for topic modeling in python
I'm using gensim in Python for topic modeling. Currently, I have one problem. If I don't lemmatize, human names will appear as 'Most Relevant Terms for Topic,' but after lemmatization, the human names ...
- 686
1
vote
1
answer
83
views
document similarity using LDA probabilities
Let us say I have a LDA model trained on a corpus of text.
I would like to know, for a newly given document, which one from the corpus is closet to it.
But, to do so, I want to use probabilities ...
- 25k
0
votes
0
answers
147
views
List of words cluster by topics
I have a list of words, these words correspond to labels in news and they are not duplicated. I would like to get a clustering for this list based on topics.
I try with wordnet but I don't know how ...
1
vote
1
answer
241
views
reuse of LDA model for new data
I am working with the LDA (Latent Dirichlet Allocation) model from sklearn and I have a question about reusing the model I have.
After training my model with data how do I use it to make a prediction ...
- 25k
25
votes
2
answers
35k
views
What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?
LDA has two hyperparameters, tuning them changes the induced topics.
What does the alpha and beta hyperparameters contribute to LDA?
How does the topic change if one or the other hyperparameters ...
CommunityBot
- 1
0
votes
0
answers
43
views
Are the word of women and men different when expressing their views on the same subject?
My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
2
votes
1
answer
758
views
extract document topic vectors from lda model
how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
CommunityBot
- 1
1
vote
1
answer
2k
views
Topic Modelling in an existing dataframe in python
I am trying to perform topic extraction in a panda dataframe. I am using LDA topic modeling in order to extract the topics in my dataframe. No problem.
But, I would like to apply LDA topic modeling ...
- 3,608
2
votes
1
answer
41
views
Topic modelling on only 24 documents gives the same "topic" for any K
Description:
I have 24 documents, each one of around 2.5K tokens. They are public speeches.
My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
- 20.5k
1
vote
0
answers
21
views
Two sets of topics/words in Topic Modeling
In short, the question is: I have two sets of words per document. I would like to extract two sets of topics per document corresponding to sets of words.
To be more precise:
Document(d) can be ...
- 11
1
vote
1
answer
78
views
Latent Dirichlet Allocation (LDA) importance of document generation and Gibbs Sampling
I am having trouble finding the correlation between the two seemingly uncorrelated parts of LDA.
What I understood from several videos is:
There is a document generation "part", which is ...
- 25k
3
votes
1
answer
163
views
How to generate synthetic text for LDA?
I am wanting to play around with LDA topic modelling, namely looking at the effects of document length, topic number etc all have on accuracy (I know it has been done elsewhere, but no one seems to ...
- 25k
1
vote
1
answer
329
views
In scikit-learn's LDA implementation, how can I sort the topics by frequency over the entire corpus?
I've used scikit-learn to perform LDA topic modeling, and I'd ultimately like to sort the topics by saliency/frequency over the entire corpus, but I'm unsure how to do as such.
I've used pyldavis ...
- 25k
1
vote
1
answer
155
views
K-means and LDA for text classification
I hope to explain in a clear way what I would like to do. I have more than 50000 tweets and I would like to add some labels on topics. So I have used LDA for doing this. I have also used k-means to ...
CommunityBot
- 1
1
vote
1
answer
87
views
What does "intractable" mean for this function in latent Dirichlet allocation (LDA)?
In the original paper Latent Dirichlet Allocation, the authors said that the function
$$p(\mathbf{w} \mid \alpha, \beta)=\frac{\Gamma\left(\sum_{i} \alpha_{i}\right)}{\prod_{i} \Gamma\left(\alpha_{i}\...
- 854
0
votes
0
answers
35
views
LSA Model Improvement
I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset.
My code is available here.
I need help with below 3 concepts.
...
- 1
0
votes
1
answer
43
views
Which tool is good to collect tweets on 50 keywords over the last 5 years and then analyze them with the LDA algorithm or sentiment analysis?
I want to find tweets from the last 5 years to a topic. For this I decide for 50 Keywords (related to the main topic), where I want to find data on Twitter. I want to find out how the trend on the ...
CommunityBot
- 1
2
votes
0
answers
78
views
What hyperparameter values does the LDA mallet model use by default? Is it true that the formula to calculate alpha = 5.0/n(topics)?
I am trying to figure out the default $\alpha$ & $\eta$ values used by mallet LDA, but there is not a lot of information on this. I did find a couple of answers, with no proper references, saying ...
- 1,635
1
vote
0
answers
30
views
Modeling of topics orthogonal to a given patterns
How to force the topics to be different from the defined ones?
Suppose I have a collection of texts about cats and dogs.There should naturally be two topics: one about dogs and one about cats. But I'm ...
- 111
2
votes
0
answers
41
views
Evaluate Topic Modelling on synthetic data
I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
- 1,012
5
votes
1
answer
993
views
Calculating optimal number of topics for topic modeling (LDA)
am going to do topic modeling via LDA. I run my commands to see the optimal number of topics. The output was as follows: It is a bit different from any other plots that I have ever seen. Do you think ...
- 1,491
1
vote
1
answer
291
views
What does updated alpha mean in LDA model?
I'm trying to understand LDA model by reading through implementations of the algorithm.
Many implementations update alpha during training iterations with codes like:
...
- 1,491
1
vote
0
answers
81
views
Best measure to indicate quality of LDA model
On my corpora, I am running LDA with different settings (I experiment with different number of topics, different different ngrams and TFIDF or regular BOW).
Now, I want to rank these setups to select ...
- 179