I am undertaking text analysis of some twitter data. In the end I want to have a data that is interpretable. And so in the end I would like to reduce the data to relevant unit of analysis. Topic models seem a nice fit since they reduce the noise significantly, but still at the end there are quite a lot of trash topic lists which do not mean anything.What would be the best way to remove these topics. Is there a working example of this?

I am using LDA. Probably I will use Gensim. Its going to be unsupervised and its a simple twitter data. So the whole corpora will be 1500 tweets of 160 characters.

  • $\begingroup$ What kind of topic model are you using? Have you tried varying the number of topics? $\endgroup$ – NBartley Aug 11 '15 at 15:03
  • $\begingroup$ What sort of analysis are you doing? Is it supervised? Is it exploratory? The issue may not even be the topics themselves but the data you are feeding into forming topics. $\endgroup$ – David Aug 11 '15 at 17:11
  • $\begingroup$ @David,@NBratley I updated my question please see above $\endgroup$ – Imo Aug 13 '15 at 19:35
  • $\begingroup$ How are you presenting the data to the model? Are you aggregating it at all? Traditional topic modeling on extremely short documents is difficult to get information out of. $\endgroup$ – NBartley Aug 14 '15 at 14:40

I assume you have inspected your topic-by-word matrix, and that is why you say you have 'trash' topics. If no words are strongly associated with a given topic, it may not be a useful topic. If you find many of these, you can re-run the experiment with fewer topics.

If instead there are words strongly associated with these topics, you just don't see the cohesive meaning behind the words associated with the topic, then these are not exactly "trash" topics. They are still capturing the latent structure of the data, it just isn't something easily deemed a "topic" in the traditional sense of the word.

Another thing to remember is that the hyperparameters also control the shape of your resulting distributions. This answer does a great job explaining alpha and beta.


I recommend calculating 1-gram, 2-grams and ... frequencies, sorting based on frequencies and observing what pops up. Sometimes you see patterns that need extra data cleansing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.