2
$\begingroup$

I am undertaking text analysis of some twitter data. In the end I want to have a data that is interpretable. And so in the end I would like to reduce the data to relevant unit of analysis. Topic models seem a nice fit since they reduce the noise significantly, but still at the end there are quite a lot of trash topic lists which do not mean anything.What would be the best way to remove these topics. Is there a working example of this?

I am using LDA. Probably I will use Gensim. Its going to be unsupervised and its a simple twitter data. So the whole corpora will be 1500 tweets of 160 characters.

$\endgroup$
4
  • $\begingroup$ What kind of topic model are you using? Have you tried varying the number of topics? $\endgroup$
    – rabbit
    Commented Aug 11, 2015 at 15:03
  • $\begingroup$ What sort of analysis are you doing? Is it supervised? Is it exploratory? The issue may not even be the topics themselves but the data you are feeding into forming topics. $\endgroup$
    – David
    Commented Aug 11, 2015 at 17:11
  • $\begingroup$ @David,@NBratley I updated my question please see above $\endgroup$
    – Imo
    Commented Aug 13, 2015 at 19:35
  • $\begingroup$ How are you presenting the data to the model? Are you aggregating it at all? Traditional topic modeling on extremely short documents is difficult to get information out of. $\endgroup$
    – rabbit
    Commented Aug 14, 2015 at 14:40

2 Answers 2

2
$\begingroup$

I assume you have inspected your topic-by-word matrix, and that is why you say you have 'trash' topics. If no words are strongly associated with a given topic, it may not be a useful topic. If you find many of these, you can re-run the experiment with fewer topics.

If instead there are words strongly associated with these topics, you just don't see the cohesive meaning behind the words associated with the topic, then these are not exactly "trash" topics. They are still capturing the latent structure of the data, it just isn't something easily deemed a "topic" in the traditional sense of the word.

Another thing to remember is that the hyperparameters also control the shape of your resulting distributions. This answer does a great job explaining alpha and beta.

$\endgroup$
0
$\begingroup$

I recommend calculating 1-gram, 2-grams and ... frequencies, sorting based on frequencies and observing what pops up. Sometimes you see patterns that need extra data cleansing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.