0
$\begingroup$

I am trying to cluster related areas of knowledge based in publications.For example a researcher has 3 keywords in a paper, in another paper he has 5 keywords, but 3 keywords are the same in the both papers. Then these 3 keywords are similar and It could be an area of knowledge. A sample of my file is:

'Domain Ontology,Semantic Web'
'Linked Data,Domain Ontology,Use Case'
'Domain Ontology,Linked Data,Semantic Annotation'
'GIS,Open GIS,Integrated Geo Systems'
'Open GIS,GIS'

My file contains 48963 rows and 19000 keywords. I have tried grouping words (like in the sample), using StringToWordVector (STWV), but I don't have good results. So I tried different K for my cluster since 3 until 13000. When K is greater my log likelihood decrease. In many models my words are grouped perfectly, but the percentages are not really good for some clusters. For example for K=3000, the 2999 have between 0,1% and 1% of the data, but the 3000 have the 21%. I am using WEKA. Anyone know some work related, some advice or any help is helpful...

Cheers.

Improve this question
$\endgroup$
7
  • $\begingroup$ These results sound typical for me, with k-means on text. Text is not what k-means was designed for, and cosine is okay for text search, but it doesn't quantiy similarity in a reliable way. $\endgroup$
    – Has QUIT--Anony-Mousse
    Oct 11, 2015 at 20:24
  • $\begingroup$ @Anony-Mousse So then Is there another reliable way to build my cluster? Or How do I estimate my true K? $\endgroup$
    – c.uent
    Oct 12, 2015 at 2:11
  • $\begingroup$ There probably is no true k. And don't expect clustering to be reliable: it is an explorative approach, not an automatic. $\endgroup$
    – Has QUIT--Anony-Mousse
    Oct 12, 2015 at 5:43
  • $\begingroup$ Yes your right! But I'm using a bag of words, so at least I could cluster similar words based in the concurrency of different papers. For example: Domain Ontology GIS Integrated Geo Systems Linked Data Open GIS Semantic Annotation Semantic Web Use Case Cluster 1 0 0 1 0 0 1 0 cluster1 1 0 0 1 0 0 0 1 cluster1 1 0 0 1 0 1 0 0 cluster1 0 1 1 0 1 0 0 0 cluster0 0 1 0 0 1 0 0 0 cluster0 Am I wrong? $\endgroup$
    – c.uent
    Oct 12, 2015 at 22:29
  • $\begingroup$ Good luck. Language when you only see bits and bytes is far from easy. I doubt 10% of your cluster assignments will be useable. $\endgroup$
    – Has QUIT--Anony-Mousse
    Oct 12, 2015 at 22:47

1 Answer 1

Reset to default
1
$\begingroup$

I don't think k-means is the correct approach to this problem. You should look at doing topic modelling using latent semantic indexing.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I read something about clustering by using latent semantic indexing, but I don't really get it! Do you know some resources to theoretical o practical? Sorry I'm new in DM and TM I'm still learning. $\endgroup$
    – c.uent
    Oct 12, 2015 at 22:25
  • 1
    $\begingroup$ No need to apologise, the best part of data analytics is that it's a continual learning process. The following link should provide a good intro >> [fastml.com/dimensionality-reduction-for-sparse-binary-data/]. Also try typing it in on Youtube to see what kind of video resources are available. $\endgroup$
    – daniel3412
    Oct 13, 2015 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.