2
$\begingroup$

Description:

I have 24 documents, each one of around 2.5K tokens. They are public speeches.

My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English contractions, removal of stopwords and tokenization.

I have implemented and analyzed both Latent Dirichlet Allocation and Latent Semantic Analysis in Python and gensim. I am calculating the optimal number of topic by the topics' coherence.

Issue:

For any number of topics K (I have tried many, e.g. 10, 50, 100, 200) I always get the same combination of top words for all topics. Therefore, they are zero informative.

I have tried removing "useless" words by threshold the TF-IDF value, but still nothing.

Diagnostic:

Trying to understand what might be the cause, I used SVD on the TF-IDF matrix. My matrix is 24 x 8115, which leads to 24 singular values. This is the plot:

enter image description here

As you can see, there is no knee point.

Maybe I can't do this since I only have 24 documents?

Or am I ignoring something fundamental for topic modelling on such a small dataset?

$\endgroup$
1
  • $\begingroup$ Can you post all of your code? There might be a bug in it. $\endgroup$ Jan 2 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.