2
$\begingroup$

Is there any python framework which takes a dataframe and gives all important relations?

For example, feature 1 and feature 2 have strong correlation when feature 3 (nominal) is equal to a special value.

$\endgroup$
8
$\begingroup$

There is a one-liner solution using a Python package called pandas-profiling that gives you a quick way into most crucial statistical explanatory analysis including various correlations and many more. The documentation provides a demo that is worth checking.

$\endgroup$
  • $\begingroup$ it's good but I'm looking for something more complicated. for example, show each important correlation with respect to each nominal value or some diagram which help to bin continuous variables. $\endgroup$ – parvij Aug 14 '18 at 6:48
  • $\begingroup$ OKay. Let us know if you find something like that. I have not found any yet. ;-) $\endgroup$ – TwinPenguins Aug 14 '18 at 7:14
5
$\begingroup$

One way to do this is calculate VIF (variance inflation factor). The feature that has the highest VIF should be removed. It is a general rule of thumb that the VIF should be less than 10. You can take a look at here if you want to try it out!

I have a dataset and I have already separated independant variable X, and dependant variable y.

X
Out[20]: 
array([[  1.  ,   7.  ,   0.27, ...,   3.  ,   0.45,   8.8 ],
       [  1.  ,   6.3 ,   0.3 , ...,   3.3 ,   0.49,   9.5 ],
       [  1.  ,   8.1 ,   0.28, ...,   3.26,   0.44,  10.1 ],
       ..., 
       [  1.  ,   6.5 ,   0.24, ...,   2.99,   0.46,   9.4 ],
       [  1.  ,   5.5 ,   0.29, ...,   3.34,   0.38,  12.8 ],
       [  1.  ,   6.  ,   0.21, ...,   3.26,   0.32,  11.8 ]])
y
Out[21]: array([6, 6, 6, ..., 6, 7, 6])

If i wanted to find out what features to remove, I would calculate the VIF as follows.

X_opt = X[:,[0,1,2,3,4,5,6,7,8,9,10,11]]

from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X_opt, i) for i in range(X_opt.shape[1])]
vif.round(1)


Out[23]: 
    VIF Factor
0    3067855.6
1          2.7
2          1.1
3          1.2
4         12.6
5          1.2
6          1.8
7          2.2
8         28.2
9          2.2
10         1.1
11         7.7

Notice that 0 has the highest VIF. So, 0 has high degree of correlation. Now, we remove it and try calculating VIF again.

X_opt = X[:,[1,3,4,5,6,7,8,9,10,11]]

vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X_opt, i) for i in range(X_opt.shape[1])]
vif.round(1)


Out[25]: 
   VIF Factor
0        92.8
1         9.7
2         3.8
3         6.4
4         8.9
5        23.7
6      1051.4
7       607.9
8        20.5
9       114.3

Now we see 6th feature has highest VIF. We continue to remove such features that have high VIF. I will leave the rest to you.

$\endgroup$
  • $\begingroup$ Welcome to the site! This VIF method seems like a good answer. You can boost your answer by not only providing an external URL, but also including an example of how to VIF in Python. $\endgroup$ – BrunoGL Aug 10 '18 at 18:57
3
$\begingroup$

The best way to quickly look for the relationships you are talking about here would be through data visualization. In particular, a correlation matrix achieves what you are looking for with regards to two variables all in one plot.

Once you find a pair of variables are correlated, you can create a 3d scatter plot using those two variables as x,y and then try all the others as z to try to detect a third relevant feature as you mentioned.

$\endgroup$
  • $\begingroup$ what about nominal and ordinal variables? $\endgroup$ – parvij Aug 14 '18 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.