1
$\begingroup$

I'm doing clustering of documents by applying k-Means on the word-vectors. To measure the cluster quality, I calculate David Bouldin Index for different k's. I tried two different distance measures, Cosine Similarity and Manhatten Distance, and get quite different values:

  • Cosine Similarity: ~0.8 to ~0.6
  • Manhattan Distance: ~0.3 to ~0.2

Can these values be compared directly? (Does Manhattan really performs a lot better here?) Or is there another way to compare the clustering results of the two different measures?

$\endgroup$
1
$\begingroup$

Comparing distance values of different distances is nonsense.

Consider this distance function:

$$d(x,y)=0$$

Clearly, this gives smaller values, but it also is useless.

Consider this trivial variation of Manhattan distance:

$$d(x,y) = \sum_i |x_i-y_i|/100$$

Clearly, this distance is equivalent to Manhattan, but will yield much smaller values.

Choose the distance because it is "the right thing to use", not because of some number. Cosine is good if you desire length normalization and have sparse vectors.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.