Questions tagged [lsi]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
28 views

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
user avatar
  • 1
2 votes
2 answers
54 views

How to represent a document in test data with the Document-Term Matrix created from the training set?

I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how ...
user avatar
2 votes
1 answer
27 views

Topic modelling on only 24 documents gives the same "topic" for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
user avatar
  • 31
1 vote
0 answers
23 views

document similarity when the document size less than 30 tokens?

I was solving a problem to compare 3 million, 2018 documents against the 2019 documents. There are three text attributes to be compared from one item against the other. I used Latent Semantic Indexing ...
user avatar
1 vote
0 answers
160 views

What is "energy spectrum" in Latent Semantic Indexing (LSI)?

What is meant by energy spectrum in LSI(Latent Semantic Indexing)? I am doing topic modeling with gensim's LsiModel, and part of the output per chunk is the following: ...
user avatar
3 votes
1 answer
115 views

Tokenizing words of length 1, what would happen if I do topic modeling?

Suppose my dataset contains some very small documents (about 20 words each). And each of them may have words in at least two languages (combination of malay and english, for instance). Also there are ...
user avatar
  • 241
2 votes
0 answers
134 views

Latent Semantic Indexing False Positive Detection

I am using the Gensim LsiModel. I have a set of documents and fixed set of topics. Some of the documents are already categorized, others are not. The goal is to categorize the uncategorized documents ...
user avatar