I have a social science background and I'm doing a text mining project. I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm analyzing a dataset of more than 200000 tweets and I'm performing a Latent Dirichlet allocation model on them to find clusters that represent the main topics of the tweets of my dataset. However, I was trying to decide the optimal number of clusters but the results I'm finding in the picture seem inconsistent. enter image description here

I'm struggling with the choice of the number of clusters. So the question is: what number would you choose from the plot? Moreover, do you think there are other ways and/or conventional rules that one can rely on to choose the number of clusters?

  • 1
    $\begingroup$ Welcome to DataScienceSE. There's no "right answer" about the number of topics in general. Quite often people manually check the top words for the different topics to decide whether the set of topics makes sense for the application. See also this related question. $\endgroup$
    – Erwan
    Sep 5 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.