Questions tagged [lda]

Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

List of words cluster by topics

I have a list of words, these words correspond to labels in news and they are not duplicated. I would like to get a clustering for this list based on topics. I try with wordnet but I don't know how ...
user avatar
1 vote
1 answer
15 views

reuse of LDA model for new data

I am working with the LDA (Latent Dirichlet Allocation) model from sklearn and I have a question about reusing the model I have. After training my model with data how do I use it to make a prediction ...
user avatar
0 votes
0 answers
18 views

Calculate an ambiguity score based on topic models and Hellinger distance

I am trying to calculate some sort of ambiguity score from text based on topic probabilities from a Latent Dirichlet Allocation model and the Hellinger distance between the topic distributions. Let’s ...
user avatar
0 votes
0 answers
40 views

Are the word of women and men different when expressing their views on the same subject?

My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
user avatar
  • 9
0 votes
0 answers
7 views

Understanding output of gensim LDA topic modeling API

I was trying to understand gensim mallet wrapper for topic modeling as explained in this notebook. In point 11, it prepares corpus which if of format Term Document frequency: ...
user avatar
  • 137
0 votes
0 answers
15 views

Deep Regression Ensembles(DRE) - text analysis

I read an article about Deep Regression Ensembles(DRE), which can outperform DNN using SDG. My question is could I use DRE in text classification? (for example, I can use it instead of LDA) What about ...
user avatar
1 vote
1 answer
182 views

Topic Modelling in an existing dataframe in python

I am trying to perform topic extraction in a panda dataframe. I am using LDA topic modeling in order to extract the topics in my dataframe. No problem. But, I would like to apply LDA topic modeling ...
user avatar
2 votes
2 answers
98 views

Topic Modeling: LDA vs LSA vs ToPMine

I am new to Topic Modeling. Is it possible to implement ToPMine in Python? In a quick search, I can't seem to find any Python package with ToPMine. Is ToPMine better than LDA and LSA? I am aware ...
user avatar
  • 159
1 vote
0 answers
15 views

Two sets of topics/words in Topic Modeling

In short, the question is: I have two sets of words per document. I would like to extract two sets of topics per document corresponding to sets of words. To be more precise: Document(d) can be ...
user avatar
  • 11
1 vote
1 answer
33 views

Latent Dirichlet Allocation (LDA) importance of document generation and Gibbs Sampling

I am having trouble finding the correlation between the two seemingly uncorrelated parts of LDA. What I understood from several videos is: There is a document generation "part", which is ...
user avatar
  • 23
0 votes
0 answers
65 views

LDA transform does produce same result on testing data as fit_transform on training data

I have a data set with over 2000 variables where I am using LDA (Linear discriminant analysis) for dimension reduction with the intention of having maximum class separability. However, LDA fails to ...
user avatar
0 votes
0 answers
12 views

Understanding the usage of variational methods in LDA

I'm reading a research paper on LDA. My goal is to understand the difference between variational methods and sampling algorithms. The specific article review that I am reading is: I would like help ...
user avatar
1 vote
1 answer
23 views

Choice of the number of topics (clusters) in textual data

I have a social science background and I'm doing a text mining project. I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm ...
user avatar
  • 11
3 votes
1 answer
52 views

How to generate synthetic text for LDA?

I am wanting to play around with LDA topic modelling, namely looking at the effects of document length, topic number etc all have on accuracy (I know it has been done elsewhere, but no one seems to ...
user avatar
0 votes
0 answers
31 views

How to interpret different coherence values

For an experiment with topic models, I have calculated four coherence values using Gensim's implementation: c_v u_mass c_uci c_npmi From this paper, I know that c_v correlates mostly with human ...
user avatar
  • 179
1 vote
1 answer
69 views

In scikit-learn's LDA implementation, how can I sort the topics by frequency over the entire corpus?

I've used scikit-learn to perform LDA topic modeling, and I'd ultimately like to sort the topics by saliency/frequency over the entire corpus, but I'm unsure how to do as such. I've used pyldavis ...
user avatar
  • 113
1 vote
1 answer
53 views

What does "intractable" mean for this function in latent Dirichlet allocation (LDA)?

In the original paper Latent Dirichlet Allocation, the authors said that the function $$p(\mathbf{w} \mid \alpha, \beta)=\frac{\Gamma\left(\sum_{i} \alpha_{i}\right)}{\prod_{i} \Gamma\left(\alpha_{i}\...
user avatar
  • 175
0 votes
0 answers
30 views

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
user avatar
  • 1
2 votes
0 answers
61 views

What hyperparameter values does the LDA mallet model use by default? Is it true that the formula to calculate alpha = 5.0/n(topics)?

I am trying to figure out the default $\alpha$ & $\eta$ values used by mallet LDA, but there is not a lot of information on this. I did find a couple of answers, with no proper references, saying ...
user avatar
1 vote
0 answers
22 views

Modeling of topics orthogonal to a given patterns

How to force the topics to be different from the defined ones? Suppose I have a collection of texts about cats and dogs.There should naturally be two topics: one about dogs and one about cats. But I'm ...
user avatar
  • 111
2 votes
0 answers
22 views

Evaluate Topic Modelling on synthetic data

I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
user avatar
5 votes
1 answer
411 views

Calculating optimal number of topics for topic modeling (LDA)

am going to do topic modeling via LDA. I run my commands to see the optimal number of topics. The output was as follows: It is a bit different from any other plots that I have ever seen. Do you think ...
user avatar
1 vote
1 answer
71 views

What does updated alpha mean in LDA model?

I'm trying to understand LDA model by reading through implementations of the algorithm. Many implementations update alpha during training iterations with codes like: ...
user avatar
1 vote
0 answers
30 views

Best measure to indicate quality of LDA model

On my corpora, I am running LDA with different settings (I experiment with different number of topics, different different ngrams and TFIDF or regular BOW). Now, I want to rank these setups to select ...
user avatar
  • 179
1 vote
0 answers
41 views

Self-supervised learning for automatic labeling of data using LDA and Word2Vec

I am trying to implement this paper A Brand-New Look at You: Predicting Brand Personality in Social Media Networks with Machine Learning for labeling Twitter data of brands with a corresponding brand ...
user avatar
  • 213
0 votes
1 answer
159 views

Topic modelling with many synonyms - how to extract 'latent themes'

Here's my corpus ...
user avatar
  • 131
0 votes
1 answer
34 views

Which algorithm should I choose and why?

My friend was reading a textbook and had this question: Suppose that you observe $(X_1,Y_1),...,(X_{100}Y_{100})$, which you assume to be i.i.d. copies of a random pair $(X,Y)$ taking values in $\...
user avatar
2 votes
0 answers
13 views

Keep retweets during topic-modelling [duplicate]

I got a dataset made out of tweets and I need to classify them into topics. For topic modelling with LDA I have cleaned out the dataset (removing stopwords, mentions, symbols, etc). Do I need to ...
user avatar
2 votes
1 answer
46 views

Tweet Classification into topics- What to do with data

Good evening, First of all, I want to apologize if the title is misleading. I have a dataset made of around 60000 tweets, their date and time as well as the username. I need to classify them into ...
user avatar
1 vote
1 answer
41 views

Are LDA clusters identical across different runs?

for a given corpus are the Latent Dirichlet Allocation clusters for it is unique in general? How about the gensim multi-process implementation of LDA? are there ...
user avatar
2 votes
0 answers
58 views

What is the objective function that Latent Dirichlet Allocation (LDA) minimizes? [closed]

Latent Dirichlet Allocation (LDA) is a generative model which produces a list of topics. Each topic is represented by a distribution over words. Questions: What is the objective function that it ...
user avatar
  • 257
1 vote
0 answers
35 views

Text Analysis : Recommendation to identify cause of loss from claim narrative documents

I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify ...
user avatar
  • 271
0 votes
1 answer
38 views

Which tool is good to collect tweets on 50 keywords over the last 5 years and then analyze them with the LDA algorithm or sentiment analysis?

I want to find tweets from the last 5 years to a topic. For this I decide for 50 Keywords (related to the main topic), where I want to find data on Twitter. I want to find out how the trend on the ...
user avatar
  • 27
2 votes
0 answers
31 views

Dealing with high dimensionality datasets

I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
user avatar
1 vote
1 answer
2k views

TF-IDF for Topic Modeling

Can TF-IDF be used a sole method for Topic Modeling ? (I know there are better methods like LDA , LSA etc) I just want to understand if TF-IDF alone can help us in Topic modeling . If yes , can ...
user avatar
  • 157
5 votes
1 answer
1k views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
user avatar
  • 151
2 votes
3 answers
107 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
user avatar
1 vote
1 answer
86 views

K-means and LDA for text classification

I hope to explain in a clear way what I would like to do. I have more than 50000 tweets and I would like to add some labels on topics. So I have used LDA for doing this. I have also used k-means to ...
user avatar
  • 151
0 votes
2 answers
300 views

k-means and LDA for text classification: how to test accuracy?

I have many tweets that I would like classify based on their similarity. Unfortunately I am not quite familiar with text-classification and nlp, so I had to read a lot of documents before having an ...
user avatar
  • 151
1 vote
1 answer
277 views

Intuition of LDA

Can anyone explain how the LDA-topic model assigns words to topics? I understand the generative property of the LDA model but how does the model recognize that "Labrador" and "dog" are similar words/ ...
user avatar
1 vote
0 answers
27 views

Topic alignment / topic modelling

What is the most efficient method for detecting whether the article is mostly about a specific topic, but without lots of data for training? My task is to determine how much a document is e.g. about ...
user avatar
  • 41
1 vote
0 answers
87 views

Trying to use term document matrix as input to orange 3

I have a CSV file that has tokens as columns and documents as rows, where the rest of the cells are ints that represent term frequency. I'm trying to use this as input into Orange 3, but Orange 3 ...
user avatar
1 vote
2 answers
179 views

Meaning of axes in Linear Discrimination Analysis

Looking at the LDA of the Iris Dataset, which looks like this: It's understandable that the 3 types of flowers have succesfuly been seperated into categories, with versicolor and virginica slightly ...
user avatar
1 vote
0 answers
91 views

How effective would this pseudo-LDA2Vec implementation be?

For my site I'm working on a chat recommender that would recommend chats to users. Each chat has a title and description and my corpus is composed of many of these title and description documents. I ...
user avatar
1 vote
0 answers
62 views

why does adding an LDA document vector with a word2vec word vector work well in LDA2vec?

In LDA the document weight vector represents the "weights" of each topic in the document. I think it's also valid to say, each row in the document vector corresponds to a word in the document, the ...
user avatar
1 vote
1 answer
109 views

How to perform topic modelling on query search results

How can I model topics in the results returned by a search engine with higher weightage to documents ranked higher in the result set? The use case that I am looking at involves extracting the most ...
user avatar
1 vote
1 answer
476 views

What does online learning mean in Topic modeling (LDA) - Gensim

I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training." So my assumption on what it means is - 'Once we have a ...
user avatar
  • 53
1 vote
2 answers
46 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
user avatar
  • 151
2 votes
1 answer
28 views

Topic modelling on only 24 documents gives the same "topic" for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
user avatar
  • 31
1 vote
0 answers
18 views

Searching for ressource recommendations (books,papers) to validate the results of a Topic Model (LDA)?

Hi I am building a Topic Model Process with Python. To do this I am using the LDA. However I am having trouble to determine the ideal amount of topics. Currently I am using the CoherenceModel ...
user avatar
  • 151