Questions tagged [lsi]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
  • 1
2 votes
2 answers

How to represent a document in test data with the Document-Term Matrix created from the training set?

I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how ...
2 votes
1 answer

Topic modelling on only 24 documents gives the same "topic" for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
  • 31
1 vote
0 answers

document similarity when the document size less than 30 tokens?

I was solving a problem to compare 3 million, 2018 documents against the 2019 documents. There are three text attributes to be compared from one item against the other. I used Latent Semantic Indexing ...
1 vote
0 answers

What is "energy spectrum" in Latent Semantic Indexing (LSI)?

What is meant by energy spectrum in LSI(Latent Semantic Indexing)? I am doing topic modeling with gensim's LsiModel, and part of the output per chunk is the following: ...
3 votes
1 answer

Tokenizing words of length 1, what would happen if I do topic modeling?

Suppose my dataset contains some very small documents (about 20 words each). And each of them may have words in at least two languages (combination of malay and english, for instance). Also there are ...
  • 241
2 votes
0 answers

Latent Semantic Indexing False Positive Detection

I am using the Gensim LsiModel. I have a set of documents and fixed set of topics. Some of the documents are already categorized, others are not. The goal is to categorize the uncategorized documents ...
user avatar