1
$\begingroup$

I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed.

Then I tried digging through the function itself, but its confusing because it can do a number of different calculations.

Docs

According to the docs:

Compute the distance matrix from a vector array X and optional Y.

I see it returns a matrix of height and width equal to the number of nested lists inputted, implying that it is comparing each one.

But otherwise I'm having a tough time understanding what its doing and where the values are coming from.

Examples I've tried:

pairwise_distances([[1]], metric='correlation')
>>> array([[0.]])

pairwise_distances([[1], [1]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

# returns same as last input although input values differ
pairwise_distances([[1], [2]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

# returns same as last input although input values differ
# I incorrectly expected more distance because input values differ more
pairwise_distances([[1,2], [1,3]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>       [2.22044605e-16, 0.00000000e+00]])

Computing correlation distance with Scipy

I don't understand where the sklearn 2.22044605e-16 value is coming from if scipy returns 0.0 for the same inputs.

# Scipy
import scipy
scipy.spatial.distance.correlation([1,2], [1,2])
>>> 0.0

# Sklearn
pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

I'm not looking for a high level explanation but an example of how the numbers are calculated.

$\endgroup$
1
$\begingroup$
Y = pdist(X, 'correlation')

Computes the correlation distance between vectors u and v. This is

$$ 1-\frac{(u-\overline{u})\dot{}(v-\overline{v})}{||u-\overline{u}||_2*||v-\overline{v}||_2} $$

where $\overline{u}$ is the mean of the elements of vector $u$, and $x\dot{}y$ is the dot product of $x$ and $y$.

The correlation between any vector which have ONLY TWO entries is always 0 (or nearly zero: $2*10^-16$), why? Because correlation distance measures the distance as the linearity between the data.

When I have [1,2] and [1,2] the equation $y=x$ fits perfectly, when I have [1,2] and [1,3] the equation $y=x+1$ also fits perfectly. The correlation distance says wheter a equation can be drawn for the data, in both cases the equation is perfect.

If you want to try getting a different result, try putting 2 vectors of three elements, and you will see changes.

Try with: [[1,2],[2,3],[3,4]] and [[1,4],[3,8],[-5,6]]. But first, plot them and you will comprehen what 'correlation' measures.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.