3
$\begingroup$

I have a data set consisting of 100 features each of which are ternary: values of -1 if it exists in one category, 0 if it doesn't exist, and 1 if it exists in the second category. For example

F1 F2 F3 ... F90 F91 F92 ... F99 F100
0  0  0  ... 1   -1  0   ... 0   -1
0  -1 0  ... -1   0  1   ... 0   0

The data is very sparse, ~20 of the 100 features have values of -1 or 1 for each row of data. I want to find similar rows of data through a heatmap visualization and dendrogram but I was confused on whether to use Euclidean distance or Cityblock distance. I'm quite new to data mining and while reading the scipy pages, I found many distance measures which I have no idea means what. Is there a good distance measure for my type of data set?

$\endgroup$

4 Answers 4

1
$\begingroup$

Since apparently each feature is encoding something about two different categories, I would suggest that you should replace that with two features. Your two features would be $(x,y)$ where $x$ is 0 or 1 according to whether it exists in the first category, and $y$ is 0 or 1 according to whether it exists in the second category. In other words, instead of -1, 0, and 1, you would use $(1,0)$, $(0,0)$, and $(0,1)$, respectively. I think that is closer to the true data and might give better results. It might also make your results easier to interpret.

Then you can try both distance metrics and see which you find more helpful -- but it might not make a large difference.

$\endgroup$
0
$\begingroup$

If you think of each row of your data as a vector a reasonable method for "distance" (similarity) would be cosine similarity. This is something that is commonly used to find similarity between user-user or item-item vectors in collaborative filtering.

$\endgroup$
0
$\begingroup$

Frankly, I don't think it matters, whether you use city-block or any generalization of Minkowski distance in this case, as long as the metric don't give different expected values when you are calculating string/vector distances. You can use city-block distance as it is computationally faster than Euclidean if you had got many combinations to calculate.

$\endgroup$
0
$\begingroup$

I think here the Hamming distance can also be considered.

In information theory, the Hamming distance between two strings of equal length is the number of positions at which the corresponding symbols are different.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.