Skip to main content
edited title
Link
James Flash
  • 301
  • 1
  • 2
  • 8

How does chi2$\chi^2$ feature selection work?

Source Link
James Flash
  • 301
  • 1
  • 2
  • 8

How does chi2 feature selection work?

I can't find the information how $\chi^2$ are used to select numerical features for a model.

Fro instance, If I employ the sklearn library:

from sklearn.datasets import load_iris
from sklearn.feature_selection import chi2

iris = load_iris()
X, y = iris.data, iris.target # X contains 4 features, y does 3 classes

I can build the following table for the dataset

pivot_table = np.zeros((X.shape[1], len(np.unique(y))))
pivot_table[:, 0] = X[y==0].sum(axis=0)
pivot_table[:, 1] = X[y==1].sum(axis=0)
pivot_table[:, 2] = X[y==2].sum(axis=0)

print(pivot_table.T)
array([[250.3, 171.4,  73.1,  12.3],
       [296.8, 138.5, 213. ,  66.3],
       [329.4, 148.7, 277.6, 101.3]])

Then applying the contingency table to it:

from scipy.stats import chi2_contingency
chi2_contingency(pivot_table, correction=False)

Output:

125.58397740261773
1.0924329599765022e-24
6
[[213.82265358 301.31664021 361.36070621]
 [111.8757204  157.6540915  189.0701881 ]
 [137.51492279 193.78458652 232.40049069]
 [ 43.88670323  61.84468177  74.168615  ]]

But this only provides the information about dependence of variables. And they are dependent because p-value is quite low.

Then I use the other function and I get what I want:

chi2(X,y)
(array([ 10.81782088,   3.7107283 , 116.31261309,  67.0483602 ]),
 array([4.47651499e-03, 1.56395980e-01, 5.53397228e-26, 2.75824965e-15]))

But how does this work? What math lies at the heart of it?