The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the general predictive power of x over y when the relationship between x and y is not linear?

  • 2
    $\begingroup$ Through information-theoretic measures like the mutual information; the reduction of entropy in one variable through knowing the other. This too can be generalized through the Renyi entropy. Welcome to the site. $\endgroup$ – Emre Apr 3 '18 at 16:39
  • $\begingroup$ That's a pretty broad question. Please edit the question to clarify what is given. Are x,y random variables? Are we given the probability distribution of x,y? Are we given a finite dataset with some values for x,y? Do you know in advance what the relationship looks like, or could it be anything? $\endgroup$ – D.W. Apr 4 '18 at 5:19

I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively called Spearman's $\rho$ and Kendall's $\tau$.

To give you an idea of what they are, consider $n$ observations from a $d$-dimensional random vector $X = (X_1,\dots,X_d)$. Also let $X_{ij}$ be the $i$th observation for variable $j$. These measures are called rank correlations because they can be computed using the ranks only. What I mean is that if you sort all $X_{ij}$, $i=1,\dots,n$, and replace the biggest observation by $n$, the second biggest by $n-1$, and so on (do that for all columns $j$) and call you new observations $R_{ij}$, then

  1. the empirical Spearman's $\rho$ (matrix) is simply the Pearson linear correlation (matrix) of $(R_1,\dots,R_d)$; and

  2. the empirical (pairwise) Kendall's $\tau$ between $X_{i_1}$ and $X_{i_2}$ is the probability of concordance minus the probability of discordance between two iid observations, say $(X_{1 i_1},X_{1 i_2})$ and $(X_{2 i_1},X_{2 i_2})$, which can equivalently be computed from the ranks $(R_{1 i_1},R_{1 i_2})$ and $(R_{2 i_1},R_{2 i_2})$ instead.

A rank correlation between $X_{i_1}$ and $X_{i_2}$ of one indeed means perfect concordance (i.e. $X_{i_1}$ always increases with $X_{i_2}$), but that does not necessarily means they are linearly related. The ranks are linearly related.

Just to make the concept of concordance clearer, here the (bivariate) observations are all concordant

Concordant observations from a bivariate random vector

and here they are all discordant

Discordant observations from a bivariate random vector

So that when you consider a cloud of points you have some pairs that are concordant and others that are discordant.

Note that this answer provides examples, but there are many other ways to approach the question. As commented by Emre, information-theoretic measures are also an option.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.