I'm working on a project and i need to discover topics existing in a social media data set. For instance, i wanna extract the topics existing on 200K tweets. Any one recommend to me any machine learning algorithms?

  • $\begingroup$ What do you mean with "topics"? Are you sure that a "topic" from topic modelling (like LDA) matches your concept of topic? Or do you want to do some clustering or labelling task? $\endgroup$
    – user10169
    Jun 20, 2016 at 12:03
  • $\begingroup$ I wanna discover topics by two methods, LDA and doing clustering or labeling tasks. $\endgroup$ Jun 20, 2016 at 14:48

3 Answers 3


You can take a look at Latent Dirichlet Allocation. In my experience this does very well without too much effort. You need to remove words that don't help like stopwords (and in your case Twitter handles and probably URLs) before feeding it to the algorithm. The only really important parameter that you need to give it is the number of topics. This will depend on your population (are these random tweets, or only tweets from a specific subgroup/hashtag?) and you need to compare some settings. What you can do is print the most important words per topic and see if they indeed do belong together.

If there are different languages in your tweets you need to deal with that beforehand, maybe classify them on language and only keep the English ones for example.

  • $\begingroup$ Thank you Jan for your reply. However, is there any other algorithms rather than using topic modeling ones ? because I was asked by my teacher to find the alternatives.. $\endgroup$ Jun 17, 2016 at 0:04

A different direction (although not necessarily a better one) is to cluster the texts you receive, perhaps using an algorithm that doesn't require many input parameters like the number of clusters. Note that not any text clustering algorithm will do - some are optimized for clustering much longer texts. This paper gives a survey of short text clustering methods: http://www.charuaggarwal.net/text-cluster.pdf It's not the newest, but it's a starting point. From experience, I agree with @Jan van der Vegt that it's recommended at the very least to look at English separately from other languages.

  • $\begingroup$ For example I clastered my data-set into N cluster. And i wanna know the topic of each one. How can I define automatically the category X from an unsupervised ML cluster. $\endgroup$ Jun 20, 2016 at 6:30
  • $\begingroup$ How about looking at the most common words or phrases from each cluster (filtering out stopwords)? $\endgroup$
    – Sharon
    Jun 20, 2016 at 7:54
  • $\begingroup$ Looking at the most common words will be like an LDA unsupervised solution that return the topics. Thus it will be more easy LDA to my twitter data-set without performing any clustering. However I'm searching the LDA alternatives, using unsupervised or supervised techniques... $\endgroup$ Jun 20, 2016 at 9:04

One approach could be the following:

1) Make a vector respresentation of all texts in the data set, for example with tfidf technique.

2) Take the first vector and put it in a pile.

3) Enter in the following loop:

3a) take the next vector and compute the cosine similarity between this vector and the centroid of each built pile.

3b) if one of this cosine similarity falls below a predefined threshold, stack this document representation in the corresponding pile. Another case, build a new pile with this vector.

3c) recompute the centroid of each modified pile.

This algorithm is going to find similar tweets, which we suppose that are related with same topic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.