# Is there any python framework which take a dataframe and give all important relation?

Is there any python framework which takes a dataframe and gives all important relations?

For example, feature 1 and feature 2 have strong correlation when feature 3 (nominal) is equal to a special value.

There is a one-liner solution using a Python package called pandas-profiling that gives you a quick way into most crucial statistical explanatory analysis including various correlations and many more. The documentation provides a demo that is worth checking.

• it's good but I'm looking for something more complicated. for example, show each important correlation with respect to each nominal value or some diagram which help to bin continuous variables. – parvij Aug 14 '18 at 6:48
• OKay. Let us know if you find something like that. I have not found any yet. ;-) – TwinPenguins Aug 14 '18 at 7:14

One way to do this is calculate VIF (variance inflation factor). The feature that has the highest VIF should be removed. It is a general rule of thumb that the VIF should be less than 10. You can take a look at here if you want to try it out!

I have a dataset and I have already separated independant variable X, and dependant variable y.

X
Out[20]:
array([[  1.  ,   7.  ,   0.27, ...,   3.  ,   0.45,   8.8 ],
[  1.  ,   6.3 ,   0.3 , ...,   3.3 ,   0.49,   9.5 ],
[  1.  ,   8.1 ,   0.28, ...,   3.26,   0.44,  10.1 ],
...,
[  1.  ,   6.5 ,   0.24, ...,   2.99,   0.46,   9.4 ],
[  1.  ,   5.5 ,   0.29, ...,   3.34,   0.38,  12.8 ],
[  1.  ,   6.  ,   0.21, ...,   3.26,   0.32,  11.8 ]])
y
Out[21]: array([6, 6, 6, ..., 6, 7, 6])


If i wanted to find out what features to remove, I would calculate the VIF as follows.

X_opt = X[:,[0,1,2,3,4,5,6,7,8,9,10,11]]

from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X_opt, i) for i in range(X_opt.shape[1])]
vif.round(1)

Out[23]:
VIF Factor
0    3067855.6
1          2.7
2          1.1
3          1.2
4         12.6
5          1.2
6          1.8
7          2.2
8         28.2
9          2.2
10         1.1
11         7.7


Notice that 0 has the highest VIF. So, 0 has high degree of correlation. Now, we remove it and try calculating VIF again.

X_opt = X[:,[1,3,4,5,6,7,8,9,10,11]]

vif = pd.DataFrame()
vif["VIF Factor"] = [variance_inflation_factor(X_opt, i) for i in range(X_opt.shape[1])]
vif.round(1)

Out[25]:
VIF Factor
0        92.8
1         9.7
2         3.8
3         6.4
4         8.9
5        23.7
6      1051.4
7       607.9
8        20.5
9       114.3


Now we see 6th feature has highest VIF. We continue to remove such features that have high VIF. I will leave the rest to you.

• Welcome to the site! This VIF method seems like a good answer. You can boost your answer by not only providing an external URL, but also including an example of how to VIF in Python. – Bruno Lubascher Aug 10 '18 at 18:57

The best way to quickly look for the relationships you are talking about here would be through data visualization. In particular, a correlation matrix achieves what you are looking for with regards to two variables all in one plot.

Once you find a pair of variables are correlated, you can create a 3d scatter plot using those two variables as x,y and then try all the others as z to try to detect a third relevant feature as you mentioned.

• what about nominal and ordinal variables? – parvij Aug 14 '18 at 6:49