Questions tagged [lda]

Latent Dirichlet Allocation (LDA) is an algorithm in the field of topic modeling.

Filter by
Sorted by
Tagged with
1
vote
0answers
14 views

Are LDA clusters identical across different runs?

for a given corpus are the Latent Dirichlet Allocation clusters for it is unique in general? How about the gensim multi-process implementation of LDA? are there ...
1
vote
0answers
15 views

What is the objective function that Latent Dirichlet Allocation (LDA) minimizes? [closed]

Latent Dirichlet Allocation (LDA) is a generative model which produces a list of topics. Each topic is represented by a distribution over words. Questions: What is the objective function that it ...
1
vote
0answers
22 views

Text Analysis : Recommendation to identify cause of loss from claim narrative documents

I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify ...
0
votes
1answer
20 views

Which tool is good to collect tweets on 50 keywords over the last 5 years and then analyze them with the LDA algorithm or sentiment analysis?

I want to find tweets from the last 5 years to a topic. For this I decide for 50 Keywords (related to the main topic), where I want to find data on Twitter. I want to find out how the trend on the ...
2
votes
0answers
23 views

Dealing with high dimensionality datasets

I have data of dimensionality (25000, 100, 500) i.e. 25000 rows each consisting of a 2 dimensional 100 X 500 matrix. Currently I am only applying CNN for ...
1
vote
1answer
65 views

TF-IDF for Topic Modeling

Can TF-IDF be used a sole method for Topic Modeling ? (I know there are better methods like LDA , LSA etc) I just want to understand if TF-IDF alone can help us in Topic modeling . If yes , can ...
3
votes
0answers
127 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
2
votes
2answers
26 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
0
votes
1answer
34 views

K-means and LDA for text classification

I hope to explain in a clear way what I would like to do. I have more than 50000 tweets and I would like to add some labels on topics. So I have used LDA for doing this. I have also used k-means to ...
0
votes
2answers
33 views

k-means and LDA for text classification: how to test accuracy?

I have many tweets that I would like classify based on their similarity. Unfortunately I am not quite familiar with text-classification and nlp, so I had to read a lot of documents before having an ...
1
vote
1answer
72 views

Intuition of LDA

Can anyone explain how the LDA-topic model assigns words to topics? I understand the generative property of the LDA model but how does the model recognize that "Labrador" and "dog" are similar words/ ...
1
vote
0answers
20 views

Topic alignment / topic modelling

What is the most efficient method for detecting whether the article is mostly about a specific topic, but without lots of data for training? My task is to determine how much a document is e.g. about ...
0
votes
0answers
54 views

Trying to use term document matrix as input to orange 3

I have a CSV file that has tokens as columns and documents as rows, where the rest of the cells are ints that represent term frequency. I'm trying to use this as input into Orange 3, but Orange 3 ...
0
votes
0answers
9 views

Distance Dependent Chinese Restaurant Process v/s LDA for document clustering

I am trying to use Distance Dependent Chinese Restaurant Process (ddCRP) for clustering documents. Specifically, clustering users in a search log (AOL log). Each user is a document with his queries ...
1
vote
2answers
35 views

Meaning of axes in Linear Discrimination Analysis

Looking at the LDA of the Iris Dataset, which looks like this: It's understandable that the 3 types of flowers have succesfuly been seperated into categories, with versicolor and virginica slightly ...
0
votes
0answers
18 views

How effective would this pseudo-LDA2Vec implementation be?

For my site I'm working on a chat recommender that would recommend chats to users. Each chat has a title and description and my corpus is composed of many of these title and description documents. I ...
0
votes
0answers
39 views

why does adding an LDA document vector with a word2vec word vector work well in LDA2vec?

In LDA the document weight vector represents the "weights" of each topic in the document. I think it's also valid to say, each row in the document vector corresponds to a word in the document, the ...
1
vote
1answer
22 views

How to perform topic modelling on query search results

How can I model topics in the results returned by a search engine with higher weightage to documents ranked higher in the result set? The use case that I am looking at involves extracting the most ...
0
votes
0answers
59 views

Topic Modelling LDA - Small dataset

I'm trying to analyze a corpus of 100 texts with the topic modelling widget of Orange using LDA algorithm. When I increase the number of topics, they repeat themselves so I have in output some ...
0
votes
1answer
85 views

What does online learning mean in Topic modeling (LDA) - Gensim

I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training." So my assumption on what it means is - 'Once we have a ...
1
vote
2answers
39 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
2
votes
0answers
16 views

Topic modelling on only 24 documents gives the same “topic” for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
0
votes
0answers
40 views

Evaluation of multi-label clustering

I am trying to find some evaluation methods for multi-label clustering. I refer to this paper: Kai Tian, Meghan Revelle, and Denys Poshyvanyk: Using Latent Dirichlet Allocation for Automatic ...
1
vote
0answers
17 views

Searching for ressource recommendations (books,papers) to validate the results of a Topic Model (LDA)?

Hi I am building a Topic Model Process with Python. To do this I am using the LDA. However I am having trouble to determine the ideal amount of topics. Currently I am using the CoherenceModel ...
0
votes
0answers
15 views

Using neural networks and user feedback for finding similar documents

I have a collection of (tens of) thousands of short documents, each containing a title and a short description of around 10-40 words. I am developing a method that finds similar/relevant documents ...
0
votes
0answers
16 views

To map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
1
vote
0answers
41 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
0
votes
1answer
67 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
0
votes
0answers
57 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
5
votes
0answers
353 views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using gensim library for topic modeling, more specifically LDA. I have created my corpus, my dictionary and my lda model, and with the help of pyLDAvis library I visualize the results. When I ...
1
vote
1answer
41 views

how to do topic modeling on very huge data?

When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
0
votes
1answer
34 views

Which are the appropriate prameters for lda modeling?

I try to implement in R test for appropriate metrics for lda. Here the way I try to use LDA ...
2
votes
1answer
108 views

Where can I learn the complete mathematics involved in LDA?

I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems. Where can I find good reading material which explains the ...
0
votes
2answers
757 views

LDA for sentiment analysis

As far as I understand it, LDA works by assuming that a corpus was written by a set of topics ands words corresponding to that topic by a specific distribution. I'm however not enterely sure what the ...
7
votes
4answers
4k views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
0
votes
0answers
50 views

Linking LDA topics to the input documents

I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
-1
votes
1answer
38 views

How can I categoriese / classify a cluster of words?

I am just wondering if it is possible to classify word clusters? For example if I provide you an array of words [bird,chicken,dock,park,apple,grapes,furits,juice] ...
0
votes
1answer
35 views

LDA as a dimensionality reducer [closed]

I know how to use LDA as a classifier. But how to use Linear Discriminant Analysis as a dimensionality reducer to reduce the number of features and apply logistic regression on top of it. I am using R ...
2
votes
1answer
729 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning ...
0
votes
2answers
329 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
1
vote
1answer
2k views

How to Combine tfidf with LSTM in keras?

I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using keras.text_to_sequences ...
2
votes
0answers
34 views

I have data of some movies and their subtitles.I want to classify them based on their ratings

I will convert the subtitles into vectors and use them as features to classify the movies into different categories based on their ratings.The problem that I am facing is my feature vector is much ...
1
vote
0answers
102 views

Model Joint Probability of N Words Appearing Together in a Sentence

Assume that we have a large corpus of texts to train with. Given N words as input, I want to model the joint probability $p(x_1, x_2, ..., x_N)$ of these words appearing together in a sentence. More ...
2
votes
1answer
168 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
2
votes
1answer
1k views

How to give names/labels to topics in LDA [duplicate]

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
2
votes
2answers
351 views

What's beyond topic modeling?

I tried topic modeling (LDA, NMF) to extract insights from the data. I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context? (...
1
vote
1answer
482 views

Industrial application(s) of LDA (latent Dirichlet allocation)?

LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining. Question What are the industrial systems using LDA or may be some related models ? (May be Google/Amazon/ ... ? ) PS I ...
1
vote
0answers
540 views

Flexible Discriminant Analysis

I am studying the book "Elements of Statistical Learning". In chapter 12 it is given about the Generalized Linear Discriminant Analysis. In one of its section it is about Flexible Discriminant ...
1
vote
1answer
629 views

How to compare the topic coherence between models of different number of topics?

If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence ...
6
votes
1answer
285 views

How to split natural language script into segments?

I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...