1
$\begingroup$

I apologize that this has been asked and I feel that it may be obvious, but I am wondering exactly what the meaning of the numerical value below from clusterdump:

Top Terms:
   monkey       =>  0.8170868432876803

I believe that to the be center of the centroid. But if the term vectors were created with term frequencies, could one interpret this as the average occurrence of the "monkey" in the documents that are considered part of the cluster? In this case, "monkey" would appear in 82% of the docs in that cluster or more likely that the average count of monkey is .82?

Looking further, I see words like so:

Top Terms: zebra => 3.432595573440644

So it is best to interpret this as the average count of "zebra" in the set of docs...

And given the radius values, one could consider that the range of percentages of "monkey"?

mahout seq2sparse -i out/sequenced \
    -o out/sparse-kmeans -wt TF --maxDFPercent 100 --namedVector

...

mahout kmeans \
    -i out/sparse-kmeans/tf-vectors/ \
    -c out/kmeans-clusters \
    -o out/kmeans \
    -dm org.apache.mahout.common.distance.CosineDistanceMeasure \
    -x 10 -k $i -ow --clustering 

When one uses tf-idf weighting, it may be best to normalize the output weights by creating a proportion of evidence via Wi=Wi/sum(W) Is that a good idea? (Some Python LDA libs do this.)

Thank you.

References

https://mahout.apache.org/users/clustering/cluster-dumper.html

$\endgroup$
5
  • $\begingroup$ Mahout k-means is useless. Good luck! $\endgroup$ Apr 2, 2016 at 14:29
  • $\begingroup$ Why do you say it is useless? $\endgroup$
    – Chris
    Apr 2, 2016 at 17:06
  • $\begingroup$ Because it is the slowest k-means I've ever seen, and it is incredibly hard to use, and it's next to impossible to get a meaningful output in the end. Yet, people expect it to do magic. $\endgroup$ Apr 2, 2016 at 19:13
  • $\begingroup$ Which library do you recommend? In regards to mahout, I have NOT noticed a method to assign clusters to subsequent files based on previous files. I wrote Python code to decipher the output! $\endgroup$
    – Chris
    Apr 3, 2016 at 1:47
  • $\begingroup$ I prefer ELKI, because it has so many methods to chose from, and it was many times faster in my experiments. $\endgroup$ Apr 3, 2016 at 8:07

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.