Questions tagged [lda]

Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

Evaluation of multi-label clustering

I am trying to find some evaluation methods for multi-label clustering. I refer to this paper: Kai Tian, Meghan Revelle, and Denys Poshyvanyk: Using Latent Dirichlet Allocation for Automatic ...
1
vote
0answers
14 views

Searching for ressource recommendations (books,papers) to validate the results of a Topic Model (LDA)?

Hi I am building a Topic Model Process with Python. To do this I am using the LDA. However I am having trouble to determine the ideal amount of topics. Currently I am using the CoherenceModel ...
0
votes
0answers
9 views

Using neural networks and user feedback for finding similar documents

I have a collection of (tens of) thousands of short documents, each containing a title and a short description of around 10-40 words. I am developing a method that finds similar/relevant documents ...
0
votes
0answers
8 views

To map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
0
votes
0answers
7 views

Differnt loglikelihood values in RandomSearch/GridSearch and LDA . How to synchronize them?

I am doing a sensitivity analysis for some lda parameters in python. This is what I want to do: -Do RandomSearch to find optimal Parameters and therefore the optimal lda model. As Validation measure ...
0
votes
0answers
18 views

Choosing a threshold based on the likelihood ratio test

I am stuck with a task of ratio test. Please help me with some advice. We are studying the Linear Discriminant Analysis. After projecting all the points on to "best" line the entire 2D dataset becomes ...
1
vote
0answers
20 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
0
votes
1answer
24 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
0
votes
0answers
17 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
0
votes
0answers
12 views

Assign Topic to a document after LDA

I executed my LDA and now I have several topics with their word distributions. How do I assign each document to a topic? Is Euclidean distance a good choice? Or there are other methods? Thanks
0
votes
0answers
21 views

Probabilistic model of selecting subsets of words from documents?

Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has ...
2
votes
0answers
48 views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using gensim library for topic modeling, more specifically LDA. I have created my corpus, my dictionary and my lda model, and with the help of pyLDAvis library I visualize the results. When I ...
0
votes
0answers
10 views

how to do topic modeling on very huge data?

When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
0
votes
0answers
10 views

Further analysis after topic identificatio

Using lda to find topics ...
0
votes
1answer
28 views

Which are the appropriate prameters for lda modeling?

I try to implement in R test for appropriate metrics for lda. Here the way I try to use LDA ...
0
votes
0answers
20 views

Class Size Imbalance for LDA or any other Content based analysis

I am running some content analysis studies on my dataset which has two different classes, and each class has a respective list of the document I am analyzing. I compare the LDA topic model inference ...
2
votes
1answer
36 views

Where can I learn the complete mathematics involved in LDA?

I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems. Where can I find good reading material which explains the ...
0
votes
2answers
171 views

LDA for sentiment analysis

As far as I understand it, LDA works by assuming that a corpus was written by a set of topics ands words corresponding to that topic by a specific distribution. I'm however not enterely sure what the ...
5
votes
2answers
865 views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
0
votes
0answers
32 views

Linking LDA topics to the input documents

I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
0
votes
0answers
17 views

how to get similary of unseen document with whole corpus

I have a trained lda model.now i want to find the new unseen document similarity with the corpus. ...
0
votes
0answers
19 views

How to Decide Topics For the Documents using LDA

I am trying to Classify Topics From Documents using LDA. I want to get topics classified as human classify from words, https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730 https:...
-1
votes
1answer
32 views

How can I categoriese / classify a cluster of words?

I am just wondering if it is possible to classify word clusters? For example if I provide you an array of words [bird,chicken,dock,park,apple,grapes,furits,juice] ...
0
votes
1answer
27 views

LDA as a dimensionality reducer [closed]

I know how to use LDA as a classifier. But how to use Linear Discriminant Analysis as a dimensionality reducer to reduce the number of features and apply logistic regression on top of it. I am using R ...
2
votes
1answer
293 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone ...
0
votes
1answer
138 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
1answer
1k views

How to Combine tfidf with LSTM in keras?

I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using keras.text_to_sequences ...
2
votes
0answers
29 views

I have data of some movies and their subtitles.I want to classify them based on their ratings

I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
1
vote
0answers
71 views

Model Joint Probability of N Words Appearing Together in a Sentence

Assume that we have a large corpus of texts to train with. Given N words as input, I want to model the joint probability $p(x_1, x_2, ..., x_N)$ of these words appearing together in a sentence. More ...
2
votes
1answer
144 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
2
votes
1answer
889 views

How to give names/labels to topics in LDA [duplicate]

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
3
votes
1answer
263 views

What's beyond topic modeling?

I tried topic modeling (LDA, NMF) to extract insights from the data. I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context? (...
1
vote
1answer
315 views

Industrial application(s) of LDA (latent Dirichlet allocation)?

LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining. Question What are the industrial systems using LDA or may be some related models ? (May be Google/Amazon/ ... ? ) PS I ...
1
vote
0answers
419 views

Flexible Discriminant Analysis

I am studying the book "Elements of Statistical Learning". In chapter 12 it is given about the Generalized Linear Discriminant Analysis. In one of its section it is about Flexible Discriminant ...
1
vote
1answer
524 views

How to compare the topic coherence between models of different number of topics?

If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence ...
5
votes
1answer
166 views

How to split natural language script into segments?

I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
4
votes
1answer
332 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
0
votes
1answer
332 views

Implementation of LDA (Latent Dirichlet Allocation) for classification tasks

Until now I have used LDA only for topic modelling. I would like to know which is the simplest implementation of LDA algorithm for classification tasks.
1
vote
1answer
36 views

Cluster algorithm to group events in more general domains

I've a list of 1,300 news events, represented by only three terms coming from running LDA topic model on thousands of tweets. Here's some of them as an example: ...
0
votes
2answers
292 views

Guided topic modeling: generating words from topics

I need to generate lists of words related to specific topics for a project. I am familiar with clustering methods of topic modeling such as LDA, but I have something else in mind. Are there any ...
1
vote
2answers
935 views

Sub topics with Latent Dirichlet Allocation

I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this. I've ...
5
votes
2answers
536 views

Topic modeling for short length sentences

I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like ...
3
votes
1answer
834 views

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
1
vote
0answers
560 views

Perplexity increasing on Test DataSet in LDA (Topic Modelling)

I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created. Unfortunately, perplexity is increasing with increased number of topics on ...
1
vote
1answer
2k views

How to map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
1
vote
1answer
169 views

LDA vs Word2Vec vs Others for predicting recipients of a message

I'm investigating various NLP algorithms and tools to solve the following problem; NLP newbie here, so pardon my question if it's too basic. Let's say, I have a messaging app where users can send ...
3
votes
1answer
110 views

stable set PCA while adding features

Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular ...
1
vote
1answer
254 views

Equally sized topics in Latent Dirichlet allocation

I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. ...
2
votes
0answers
1k views

why the accuracy of LDA model is always changing and also is high

Let’s explain the whole goal firstly, then go through the question. I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents. My dataset ...
2
votes
0answers
1k views

fitting classifier object of type 'int' has no len()

We have LDA topic modeling whose purpose is to generate a number of topics given a set of documents. So each document can belong to various topics. Also, we can evaluate the model we have created. ...