1
$\begingroup$

I have a dataset which looks like this-

enter image description here

I am trying to use chi2 as a feature selction algorithm on it.Here is the code

dataset = pd.read_csv('small_dataset.csv')
print(dataset.shape)


X = dataset.iloc[:, 1:106].values
y = dataset.iloc[:,0].values


print(sklearn.feature_selection.chi2(X, y))

I tried to validate the answers from sklearn and the formula which is

enter image description here

enter image description here

so I took the first feature and computed the A B C D which are 0 19 73 73 and N is 146.So the X2 value coming is 21 but in sklearn its coming 579.

What am I doing wrong?

$\endgroup$
1
  • $\begingroup$ Did I answer it? $\endgroup$ Jun 14 '18 at 7:31
1
$\begingroup$

I had a similar problem using Scipy's chi2_contingency. It turns out that the Scipy function uses by default correction=True, and the statistic you showed doesn't use the correction. Perhaps sklearn's chi2 uses the correction as well, and that's the reason you don't get the same results. I suggest trying Scipy's with both correction set to true and false to see if it matches the two results.

$\endgroup$
1
  • $\begingroup$ the problem with scipy is it doesnt accept as input the feature vector table I have as a input....thats why I used the skearn $\endgroup$ Jun 14 '18 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.