I am trying to cluster related areas of knowledge based in publications.For example a researcher has 3 keywords in a paper, in another paper he has 5 keywords, but 3 keywords are the same in the both papers. Then these 3 keywords are similar and It could be an area of knowledge. A sample of my file is:

'Domain Ontology,Semantic Web'
'Linked Data,Domain Ontology,Use Case'
'Domain Ontology,Linked Data,Semantic Annotation'
'GIS,Open GIS,Integrated Geo Systems'
'Open GIS,GIS'

My file contains 48963 rows and 19000 keywords. I have tried grouping words (like in the sample), using StringToWordVector (STWV), but I don't have good results. So I tried different K for my cluster since 3 until 13000. When K is greater my log likelihood decrease. In many models my words are grouped perfectly, but the percentages are not really good for some clusters. For example for K=3000, the 2999 have between 0,1% and 1% of the data, but the 3000 have the 21%. I am using WEKA. Anyone know some work related, some advice or any help is helpful...


  • $\begingroup$ These results sound typical for me, with k-means on text. Text is not what k-means was designed for, and cosine is okay for text search, but it doesn't quantiy similarity in a reliable way. $\endgroup$ – Has QUIT--Anony-Mousse Oct 11 '15 at 20:24
  • $\begingroup$ @Anony-Mousse So then Is there another reliable way to build my cluster? Or How do I estimate my true K? $\endgroup$ – c.uent Oct 12 '15 at 2:11
  • $\begingroup$ There probably is no true k. And don't expect clustering to be reliable: it is an explorative approach, not an automatic. $\endgroup$ – Has QUIT--Anony-Mousse Oct 12 '15 at 5:43
  • $\begingroup$ Yes your right! But I'm using a bag of words, so at least I could cluster similar words based in the concurrency of different papers. For example: Domain Ontology GIS Integrated Geo Systems Linked Data Open GIS Semantic Annotation Semantic Web Use Case Cluster 1 0 0 1 0 0 1 0 cluster1 1 0 0 1 0 0 0 1 cluster1 1 0 0 1 0 1 0 0 cluster1 0 1 1 0 1 0 0 0 cluster0 0 1 0 0 1 0 0 0 cluster0 Am I wrong? $\endgroup$ – c.uent Oct 12 '15 at 22:29
  • $\begingroup$ Good luck. Language when you only see bits and bytes is far from easy. I doubt 10% of your cluster assignments will be useable. $\endgroup$ – Has QUIT--Anony-Mousse Oct 12 '15 at 22:47

I don't think k-means is the correct approach to this problem. You should look at doing topic modelling using latent semantic indexing.

  • $\begingroup$ I read something about clustering by using latent semantic indexing, but I don't really get it! Do you know some resources to theoretical o practical? Sorry I'm new in DM and TM I'm still learning. $\endgroup$ – c.uent Oct 12 '15 at 22:25
  • 1
    $\begingroup$ No need to apologise, the best part of data analytics is that it's a continual learning process. The following link should provide a good intro >> [fastml.com/dimensionality-reduction-for-sparse-binary-data/]. Also try typing it in on Youtube to see what kind of video resources are available. $\endgroup$ – daniel3412 Oct 13 '15 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.