Questions tagged [lda]
Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.
49
questions with no upvoted or accepted answers
8
votes
1
answer
1k
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
- 81
5
votes
1
answer
2k
views
How to choose threshold for gensim Phrases when generating bigrams?
I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA
...
- 151
3
votes
0
answers
2k
views
why the accuracy of LDA model is always changing and also is high
Let’s explain the whole goal firstly, then go through the question.
I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents.
My dataset ...
- 331
2
votes
0
answers
78
views
What hyperparameter values does the LDA mallet model use by default? Is it true that the formula to calculate alpha = 5.0/n(topics)?
I am trying to figure out the default $\alpha$ & $\eta$ values used by mallet LDA, but there is not a lot of information on this. I did find a couple of answers, with no proper references, saying ...
2
votes
0
answers
40
views
Evaluate Topic Modelling on synthetic data
I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
- 21
2
votes
1
answer
45
views
Dealing with high dimensionality datasets
I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
- 41
2
votes
3
answers
227
views
How to identify text similarity based on training data?
I have a set of documents (1 to 11) for which the labeling is done.
Lets Assume:
...
- 129
2
votes
1
answer
40
views
Topic modelling on only 24 documents gives the same "topic" for any K
Description:
I have 24 documents, each one of around 2.5K tokens. They are public speeches.
My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
- 31
2
votes
0
answers
89
views
Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?
I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
- 131
2
votes
0
answers
39
views
I have data of some movies and their subtitles.I want to classify them based on their ratings
I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
- 107
2
votes
0
answers
1k
views
fitting classifier object of type 'int' has no len()
We have LDA topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics.
Also, we can evaluate the model we have created. one ...
- 331
2
votes
0
answers
9k
views
Found array with dim 3. Estimator expected <= 2
I am using LDA over a simple collection of documents. My goal is to extract topics, then use the extracted topics as features to evaluate my model.
I decided to use multinomial SVM as the evaluator.
...
- 331
2
votes
1
answer
2k
views
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
I can not seem to find it anywhere, even not from the author of LDA, ...
- 55
2
votes
0
answers
132
views
Extending Author-Topic LDA
I've been trying to extend the LDA and wanted some help, direction and insight.
Can Author-Topic LDA be used as a document "category" model? The premise of the Author-Topic model is that multiple ...
- 21
1
vote
0
answers
171
views
Short Text Topic Modelling in Python
I have a large dataset of short reviews and I would like to find the most recurring themes. For this reason, I got into topic modeling.
I am looking for some good tutorials and references for short ...
1
vote
0
answers
20
views
Two sets of topics/words in Topic Modeling
In short, the question is: I have two sets of words per document. I would like to extract two sets of topics per document corresponding to sets of words.
To be more precise:
Document(d) can be ...
- 11
1
vote
1
answer
83
views
Choice of the number of topics (clusters) in textual data
I have a social science background and I'm doing a text mining project.
I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm ...
- 11
1
vote
0
answers
30
views
Modeling of topics orthogonal to a given patterns
How to force the topics to be different from the defined ones?
Suppose I have a collection of texts about cats and dogs.There should naturally be two topics: one about dogs and one about cats. But I'm ...
- 111
1
vote
0
answers
79
views
Best measure to indicate quality of LDA model
On my corpora, I am running LDA with different settings (I experiment with different number of topics, different different ngrams and TFIDF or regular BOW).
Now, I want to rank these setups to select ...
- 179
1
vote
0
answers
51
views
Self-supervised learning for automatic labeling of data using LDA and Word2Vec
I am trying to implement this paper A Brand-New Look at You: Predicting Brand Personality in Social Media Networks with Machine Learning for labeling Twitter data of brands with a corresponding brand ...
- 213
1
vote
0
answers
43
views
Text Analysis : Recommendation to identify cause of loss from claim narrative documents
I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify ...
- 281
1
vote
1
answer
121
views
Topic alignment / topic modelling
What is the most efficient method for detecting whether the article is mostly about a specific topic, but without lots of data for training? My task is to determine how much a document is e.g. about ...
- 51
1
vote
0
answers
111
views
Trying to use term document matrix as input to orange 3
I have a CSV file that has tokens as columns and documents as rows, where the rest of the cells are ints that represent term frequency. I'm trying to use this as input into Orange 3, but Orange 3 ...
- 61
1
vote
0
answers
110
views
How effective would this pseudo-LDA2Vec implementation be?
For my site I'm working on a chat recommender that would recommend chats to users. Each chat has a title and description and my corpus is composed of many of these title and description documents. I ...
1
vote
0
answers
82
views
why does adding an LDA document vector with a word2vec word vector work well in LDA2vec?
In LDA the document weight vector represents the "weights" of each topic in the document. I think it's also valid to say, each row in the document vector corresponds to a word in the document, the ...
1
vote
0
answers
18
views
Searching for ressource recommendations (books,papers) to validate the results of a Topic Model (LDA)?
Hi I am building a Topic Model Process with Python.
To do this I am using the LDA.
However I am having trouble to determine the ideal amount of topics.
Currently I am using the CoherenceModel ...
- 151
1
vote
0
answers
214
views
Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?
If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
- 131
1
vote
0
answers
185
views
Model Joint Probability of N Words Appearing Together in a Sentence
Assume that we have a large corpus of texts to train with. Given N words as input, I want to model the joint probability $p(x_1, x_2, ..., x_N)$ of these words appearing together in a sentence. More ...
- 11
1
vote
0
answers
799
views
Flexible Discriminant Analysis
I am studying the book "Elements of Statistical Learning". In chapter 12 it is given about the Generalized Linear Discriminant Analysis. In one of its section it is about Flexible Discriminant ...
- 131
1
vote
1
answer
411
views
Implementation of LDA (Latent Dirichlet Allocation) for classification tasks
Until now I have used LDA only for topic modelling. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.
- 695
1
vote
0
answers
978
views
Perplexity increasing on Test DataSet in LDA (Topic Modelling)
I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created.
Unfortunately, perplexity is increasing with increased number of topics on ...
- 111
1
vote
1
answer
3k
views
How to map topic to a document after topic modeling is done with LDA
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ?
I am interested in clustering documents using unsupervised learning and ...
- 19
1
vote
0
answers
272
views
Scikit Learn Latent Dirichlet Allocation overload my SWAP and/or RAM
1) After using LDA of scikit learn I realized it overloads the swap, but never reach more than 20% of the RAM. Do you have any idea why ? I can't really see if this is a problem, but still I would ...
- 1,307
0
votes
0
answers
27
views
number of topics in LDA model (coherence)
I want to fit a LDA model using RStudio, but there are some trouble in the determination of topics number.
Perplexity and coherence are suggested to do this work, so first I use LDA and perplexity ...
- 1
0
votes
0
answers
6
views
Is colorring each document and word as monochromatic as possible the goal of LDA in specific or all topic models in general?
In the video Training Latent Dirichlet Allocation: Gibbs Sampling, the Goal section at 7:32, the video says that:
Goal: Color each word with blue, green, red
Each article is as monochromatic as ...
- 101
0
votes
0
answers
36
views
Doing LDA (NLP task): Does it make sense to use tf-idf vectors?
Most implementations that I've seen of LDA seem to use simple word counts when giving the document-term frequency matrix.
What would happen if we were to give the tf-idf matrix instead of a simple ...
0
votes
0
answers
20
views
How the embedding model (x-vectors) trained?
I read this paper: X-Vectors: Robust DNN Embeddings for Speaker Recognition which describes how PyAnnote embedding block works.
I'm not sure I understand how the X-Vector model was trained and tested:
...
- 363
0
votes
0
answers
21
views
LDA calculations manually
my question: has anyone ever done LDA calculations manually? I have difficulty in manual calculation. can someone help me to teach me for lda calculations manually.
- 1
0
votes
0
answers
20
views
Algorithm of lda2vec in NLP
I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
- 1
0
votes
1
answer
16
views
method of allocating documents to pre-defined classifications
I'm looking for a method of allocating documents (30K and growing) to a set of some 200 categories.
The categories will be user defined and will grow over time.
As my data is unlabelled my thought ...
- 101
0
votes
0
answers
9
views
Get top documents per topic from mallet lda
I'm using java mallet's LDA engine to determine topics for a set of (some 30K) documents.
I've managed to train the model and serialize/deserialize it as needed.
The question is how do I find the top '...
- 101
0
votes
0
answers
146
views
List of words cluster by topics
I have a list of words, these words correspond to labels in news and they are not duplicated. I would like to get a clustering for this list based on topics.
I try with wordnet but I don't know how ...
0
votes
0
answers
43
views
Are the word of women and men different when expressing their views on the same subject?
My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
- 9
0
votes
0
answers
35
views
LSA Model Improvement
I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset.
My code is available here.
I need help with below 3 concepts.
...
- 1
0
votes
1
answer
353
views
0
votes
0
answers
84
views
To map topic to a document after topic modeling is done with LDA
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
- 11
0
votes
0
answers
70
views
Linking LDA topics to the input documents
I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
- 1
0
votes
2
answers
698
views
Apply Labeled LDA on large data
I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
-1
votes
1
answer
2k
views
Words from LDA output pyspark machine learning
I built a pipeline for an LDA model using pyspark's machine learning.
Here is my code:
...
