Questions tagged [topic-model]

A topic model describes text from a large corpus as a probability distribution over topics which are probability distributions over words. There are quantified contributions from all topics to a specific text.

Filter by
Sorted by
Tagged with
0
votes
0answers
54 views

Linking LDA topics to the input documents

I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
1
vote
1answer
41 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
1
vote
0answers
13 views

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python

How to find out the subject of an email (in the form of a sentence) or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic ...
2
votes
1answer
955 views

Online vs Batch Learning in Latent Dirichlet Allocation using Scikit Learn

Reference I'm looking at the LDA algorithm from Scikit Learn for topic modeling. Can someone tell me how the 'online' method of learning works vs the 'batch' method of learning? Also, what is learning ...
5
votes
1answer
1k views

NLP algorithms for categorizing a list of words with specific topics

Currently I am using LDA to apply topic modeling to a corpus. Since LDA is unsupervised, it returns a set of words for a given 'topic' but doesn't necessarily specify the topic itself. I was wondering ...
0
votes
1answer
469 views

Is it correct to create topic models using both train and test data?

I have a dataset of text documents splitted into train and test sets. My task is a binary classification, classifying these documents to either 1 or -1. I have already computed some features using TF-...
0
votes
1answer
64 views

Topic Segmentation - should it be done in Raw, TfIdf or Semantic Space?

Let's assume we have a collection of documents and wish to perform some unsupervised topic segmentation. As always, we will perform some preprocessing (including tokenization, accent-removal, ...
2
votes
1answer
2k views

How to give names/labels to topics in LDA [duplicate]

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
2
votes
2answers
366 views

What's beyond topic modeling?

I tried topic modeling (LDA, NMF) to extract insights from the data. I'm curious right now, are there other methods for unsupervised learning to cluster documents by the same or similar context? (...
8
votes
1answer
21k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
3
votes
1answer
68 views

Evaluating the result of topic modeling in a way that time matters

I have run different topic modeling approach on my data(its clinical data related to Cognitive impairment diseases. we are going to process what thing is important that make it develop to more harsh ...
1
vote
1answer
46 views

How to build News Tagging model(s)

I am trying to build a news tagging system. Given a piece of news article, find 5-6 key terms from the news article that best describe the article. Refer to the image below from google news. What are ...
1
vote
1answer
660 views

How to compare the topic coherence between models of different number of topics?

If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence ...
6
votes
1answer
309 views

How to split natural language script into segments?

I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the ...
4
votes
1answer
451 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
0
votes
2answers
308 views

Guided topic modeling: generating words from topics

I need to generate lists of words related to specific topics for a project. I am familiar with clustering methods of topic modeling such as LDA, but I have something else in mind. Are there any ...
1
vote
0answers
103 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
3
votes
2answers
107 views

Collection Of Variable Length Sequences and Descriptions: A Search Problem

I have a tough problem and need some advice: Suppose I have a collection of variable length sequences, many of which are unique -- imagine the moves to a chess game, eg d4 Nf6 c4 g6 Nc3 Bg7 ...
10
votes
1answer
5k views

What is the difference between topic modeling and clustering?

I know that topic modeling and clustering are related, but not similar techniques. Can anyone suggest what are the main differences?
0
votes
1answer
465 views

Automatic question categorization when we know important words in each category

I am currently working on a question categorization problem where I automatically want to assign a category to the question. The question set I have is unlabelled. The categories for the problem are ...
1
vote
0answers
724 views

Perplexity increasing on Test DataSet in LDA (Topic Modelling)

I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created. Unfortunately, perplexity is increasing with increased number of topics on ...
1
vote
3answers
517 views

What are key dataset requirements for topic models and word embeddings?

I have a dataset of 2000 documents where avg doc size is 300 words. The vocab is dominated by domain-specific words. My goal is to find similar documents. For this, I tried LDA, LSI, Doc2Vec (topics=...
1
vote
1answer
2k views

How to map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
4
votes
2answers
1k views

Compare two topic modelling sets

I have two sets of newspaper articles where I train the first newspaper dataset separately to get the topics per each newspaper article. ...
1
vote
1answer
370 views

Hellinger Distance in Gensim

I have set of documents as follows where each document has set of words that represents the content of it. ...
5
votes
1answer
883 views

Combine two sets of clusters

I have two sets of topics obtained from two different sets of news paper articles. In other words, Cluster_1 = ${x_1, x_2, ..., x_n}$ includes the main topics of 'X' news paper set and Cluster_2 = ${...
1
vote
1answer
370 views

Equally sized topics in Latent Dirichlet allocation

I'm using the topicmodels package for R to cluster a big set of short texts (between 10-75 words) into topics. After manually reviewing a few models it seems like there are 20 realtivly stable topics. ...
3
votes
0answers
1k views

why the accuracy of LDA model is always changing and also is high

Let’s explain the whole goal firstly, then go through the question. I am using topic modeling like LAtent Dirichlet Allocation and NMF to extract the topic from a collection of documents. My dataset ...
9
votes
2answers
14k views

Why we should not feed LDA with tfidf

Can someone explain why we can not feed LDA topic model with TFIDF? What is wrong with this approach conceptually?
2
votes
1answer
3k views

Apply SVM on LDA in python

hope someone kindly put time here, my approach is like this: TFIDF -> LDA -> SVM I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document ...
1
vote
1answer
631 views

News topic detection and categorization

If I want to get how many and what kind of topics are covered by New York Times each week from a bag of words model(All the news covered by NYT in a week) how should I approach? Using traditional ...
1
vote
0answers
156 views

What is “energy spectrum” in Latent Semantic Indexing (LSI)?

What is meant by energy spectrum in LSI(Latent Semantic Indexing)? I am doing topic modeling with gensim's LsiModel, and part of the output per chunk is the following: ...
0
votes
1answer
3k views

How to use Non Negative Matric Factorization (NMF)'s transform method to project topics on new text data

I have created a NMF topic model in python the code snippet for which is as follows: ...
0
votes
1answer
272 views

Quantifying the Reproducibility of LDA Models

I am working on a text mining project where I'm using Latent Dirichlet Allocation to study a corpus of documents. I'm currently in the process of optimizing my parameters to get the best models for my ...
1
vote
1answer
500 views

Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities

Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document. I know that I have to compute the ...
0
votes
1answer
295 views

i have to classify an sms into categories like educational, bank related,etc. Is this a problem of topic modelling or text classification? [closed]

If this is a problem of text classification, is a similar dataset available or i have to make one on my own?? I have a dataset of emails for classification into spam/ham.
6
votes
1answer
715 views

Comparing two Corpora using Topic Model

I want to compare two corpora (two different collections of texts) using Topic Modeling. I trained the model separately on the two collections and manually matched similar topics based on their ...
0
votes
1answer
2k views

Goodness of fit metric to compare topic models NMF vs LDA

Does anyone have a good idea for how to compare topic modeling done by NMF and LDA? Let's say I fit LDA to a dataset and generate topic-word and document-topic distributions--I can use perplexity, for ...
24
votes
1answer
7k views

NLP - why is “not” a stop word?

I am trying to remove stop words before performing topic modeling. I noticed that some negation words (not, nor, never, none etc..) are usually considered to be stop words. For example, NLTK, spacy ...
4
votes
2answers
6k views

NLTK Sklearn Genism Text to Topic

I aint no data scientist/machine learner. What Im Lookin for ...
3
votes
3answers
2k views

scikit-learn - Should I fit model with TF or TF-IDF?

I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python). Looking at the example in ...
2
votes
1answer
264 views

How to get columns from unsorted rows in Pandas? (MALLET)

My data (the doc-topics output from a MALLET topic model) has the following shape: ...
-1
votes
1answer
131 views

Topic modelling or simple case of calculating probabilities?

I am trying to find the common topics between articles read using the respective tags attached to each article. Background of my mini project: The problem I am trying to solve involves looking at ...
1
vote
0answers
74 views

automated topic modeling topic naming [duplicate]

Are there well-known automated methods for deriving a name for each topic obtained through topic modeling? for a specifically given problem at hand I will probably default to an algorithm on top an ...
1
vote
1answer
106 views

Is what I did supervised or unsupervised Machine learning?

My goal is to get a smartphone names from Twitter. So this is what I followed: 1- I extracted 100K tweets using the keyword “smartphone”. 2- I Applied LDA after applying ngram tokenization and ...
4
votes
1answer
2k views

How can I run Labeled LDA over one textual document?

I have 200K tweets and I already a applied the LDA (Latent Dirichlet Allocation) algorithm using Gensim python library. And now I need to apply over them the labeled/supervised LDA. Can any one help ...
1
vote
1answer
2k views

Real time topic identification of news article

Let's say I'm constantly harvesting all the news article that are being published online (only having basic info about each one, eg. title, content, language, source (which news site)). Let's say ...
4
votes
3answers
150 views

How can I discover topics in a social media data-set?

I'm working on a project and i need to discover topics existing in a social media data set. For instance, i wanna extract the topics existing on 200K tweets. Any one recommend to me any machine ...
-1
votes
1answer
642 views

Using Topic Models in R

I am learning about Probabilistic Topic Models by reading this article by D. Blei, watching this video, and doing this exercise A Gentle Introduction to Topic Modeling in R. After the topics in my ...
1
vote
2answers
444 views

Memory error - Hierarchical Dirichlet Process, HDP gensim

I am running Hierarchical Dirichlet Process, HDP using gensim in Python but as my corpus is too large it is throwing me following error: ...