7
$\begingroup$

Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{0,1\}$ or one of them can be continuous. I would like to perform some statistical test to be sure they are independent. I am using Python so it would nice to have some ready-to-use tool implemented. I my also use R if it is not something difficult to do.

$\endgroup$
3
$\begingroup$

As is discussed in the link to Cross-Validated SO from Mephy, this is isn't an easy thing to do.

If they are independent, you might expect a correlation between pairs of variables to be close to zero. That would mean that knowing anything about one of the two variables doesn't give you any insight as to the behaviour of the second. To this end, there is a nice answer here, which shows how to compute pairwise pearson correlation (with corresponding p-values) for all columns in a Pandas DataFrame.

The Pearson correlation does assume your random variables to be normally distributed, so keep that in mind when interpreting results. Alternatively, you could swap out that pearsonr function for the Spearman Rank correlation function: spearmanr, which does not assume normality of your variables.

Another (perhaps simpler) way using just a Pandas DataFrame is to use the built in method corr: This takes a keyword method, which allows you to specify one of three:

method : {‘pearson’, ‘kendall’, ‘spearman’}


If you random variables are time-series (you didn't mention it), another possible tool to look at would be Granger Causality. This could also be performed pairwise (or batch-wise) across variables. It tests to see if a the future value of variable can be better predicted when historic values of the a different variable are included in the model. For example, if the price of StockA can be predicted with an accuracy of 52% using its own prices of the previous 5 days, the Granger test would have a null hypothesis that including some lags from StockB would not improve the accuracy. So if the accuracy does indeed jump up to 53% when including lagged prices of StockB, (and the test is significant), the null hypothesis is rejected and we say that StockB Granger-causes StockA.

This is implemented in the vars package in R (there are others too). As a bonus, this version can also perform the Wald test for correlation in error processes of predictor and target variables.

$\endgroup$
1
$\begingroup$

This is a hard problem to solve and there are many tests that attempt to answer it.. One way of testing it is using mutual information, available on scikit-learn for either continuous variables or discrete variables. It returns zero for independent variables and higher values the more dependence there is between the variables (makes it harder to call something "independent/dependent" but makes it easier to rank features by their independence).

from sklearn.feature_selection import mutual_info_regression
import numpy as np

x = np.linspace(0, 10, 50)
y = x + np.random.randn(50)
z = np.random.randn(50)

# reshape necessary because the function accepts many
# features at once to be compared with the right-hand side
print(mutual_info_regression(x.reshape(-1, 1), y))
print(mutual_info_regression(x.reshape(-1, 1), z))

> 1.20832658
> 0
$\endgroup$
0
$\begingroup$

For the sake of those wanderers looking for more info about testing for independence, conditional in this case, I'm referencing a paper I read recently, despite the fact that it may not be exactly what the OP is looking for.

An interesting approach is presented by researchers from CalTech where they use a decision tree to test for conditional independence here.

the corresponding repo is here.

One apparent advantage of this method as compared to others is its speed. I say apparent as I've yet to try it. The benchmarking included in the paper, nonetheless, are impressive.

$\endgroup$
1
  • $\begingroup$ Unforetunately fcit doesn't seem to be as fast as it's claims, even with tiny dimensions. Hopefully we can take another look at it as an option after the next update. $\endgroup$ – Daniel Scott Apr 28 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.