Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am working on a clustering problem. I am not able to find the right similarity metric for my system.
I have n nodes with ordered vector (eg: [1,0,0,0,1,0,.....] "1" represents represent presence of ith object and "0" represents absence of the object). I want to cluster them.
Cosine is for continuous values. It's not the most appropriate thing here.
For binary values, look at
Don't expect clustering to just work. If you have binary data, you only have very few bits of information. The resulting distances will show discrete steps, and clustering will likely still fail to provide good results. Roughly, you have "identical", "one difference", "two differences", and then probably everything already is connected. It's because of the data, and a not very well defined task (what would be a good cluster?).
Cosine similarity is a popular choice: https://en.wikipedia.org/wiki/Cosine_similarity
As you can imagine, the answer is "it depends".
But in your case you can choose between two: Euclidean or Cosine.
As the dimensionality grows every point approach the border of the multi dimensional space where they lie, so the Euclidean distances between points tends asymptotically to be the same, which in similarity terms means that the points are all very similar to each other.
So, in those cases, you go for the cosine similarity; let's say for dimensionality bigger than 100.
Otherwise you can go for the traditional, Euclidean similarity.
I would strongly recommend to also have a look at Graph Theoretic measures as your problem can be easily and properly seen as a Bipartite-Graph Clustering.
