# What does sklearn's pairwise_distances with metric='correlation' do?

I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed.

Then I tried digging through the function itself, but its confusing because it can do a number of different calculations.

Docs

According to the docs:

Compute the distance matrix from a vector array X and optional Y.


I see it returns a matrix of height and width equal to the number of nested lists inputted, implying that it is comparing each one.

But otherwise I'm having a tough time understanding what its doing and where the values are coming from.

Examples I've tried:

pairwise_distances([[1]], metric='correlation')
>>> array([[0.]])

pairwise_distances([[1], [1]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

# returns same as last input although input values differ
pairwise_distances([[1], [2]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

# returns same as last input although input values differ
# I incorrectly expected more distance because input values differ more
pairwise_distances([[1,2], [1,3]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>       [2.22044605e-16, 0.00000000e+00]])



Computing correlation distance with Scipy

I don't understand where the sklearn 2.22044605e-16 value is coming from if scipy returns 0.0 for the same inputs.

# Scipy
import scipy
scipy.spatial.distance.correlation([1,2], [1,2])
>>> 0.0

# Sklearn
pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])


I'm not looking for a high level explanation but an example of how the numbers are calculated.

Y = pdist(X, 'correlation')


Computes the correlation distance between vectors u and v. This is

$$1-\frac{(u-\overline{u})\dot{}(v-\overline{v})}{||u-\overline{u}||_2*||v-\overline{v}||_2}$$

where $$\overline{u}$$ is the mean of the elements of vector $$u$$, and $$x\dot{}y$$ is the dot product of $$x$$ and $$y$$.

The correlation between any vector which have ONLY TWO entries is always 0 (or nearly zero: $$2*10^-16$$), why? Because correlation distance measures the distance as the linearity between the data.

When I have [1,2] and [1,2] the equation $$y=x$$ fits perfectly, when I have [1,2] and [1,3] the equation $$y=x+1$$ also fits perfectly. The correlation distance says wheter a equation can be drawn for the data, in both cases the equation is perfect.

If you want to try getting a different result, try putting 2 vectors of three elements, and you will see changes.

Try with: [[1,2],[2,3],[3,4]] and [[1,4],[3,8],[-5,6]]. But first, plot them and you will comprehen what 'correlation' measures.