2
$\begingroup$

i’am a beginner in scikit-learn and i’ve a little problem when using feature selection module VarianceThreshold, the problem is when i set the variance Var[X]=.8*(1-.8)

it is supposed to remove all features (that have the same value in all samples) which have the probability $p>0.8$. in my case the fifth column should be removed, p=8/10>(threshold=0,7).

#####################################

from sklearn.feature_selection import VarianceThreshold
X=[[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.01,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00],
[0,1,2,1,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0.00,0.00,0.00,0.00,0.50,1.00,0.00,10,3,0.30,0.30,0.30,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,253,0.99,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,2,3,1,223,185,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,4,4,0.00,0.00,0.00,0.00,1.00,0.00,0.00,71,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00]]
sel=VarianceThreshold(threshold=(.7*(1-.7)))

and this is what i get when running the script

>>> sel.fit_transform(X)

array([[ 1., 105., 146., 1., 1., 255., 254.],
[ 1., 105., 146., 1., 1., 255., 254.],
[ 1., 105., 146., 1., 1., 255., 254.],
[ 1., 105., 146., 2., 2., 255., 254.],
[ 1., 105., 146., 2., 2., 255., 254.],
[ 1., 105., 146., 2., 2., 255., 255.],
[ 2., 29., 0., 2., 1., 10., 3.],
[ 1., 105., 146., 1., 1., 255., 253.],
[ 1., 105., 146., 2., 2., 255., 254.],
[ 3., 223., 185., 4., 4., 71., 255.]])

the second column here should not apear. thanks;)

$\endgroup$
1
  • $\begingroup$ This logic only works for Bernoulli distributed features (ones and zeros, where probability for a 1 is p, and then the variance is given as p*(1-p)). Your fifth/sixth column is clearly not Bernoulli distributed. $\endgroup$
    – oW_
    Jun 13 '17 at 15:34
0
$\begingroup$

0.8 is the threshold value meaning you want to exclude columns(features) with at least 80% of values is 0 or 1: sort of unbalanced dummies. This is only applied well to dummies, but not necessarily general numerical values. For numerical, you can test RFECV function in feature_selection package from sciki-learn, the syntax as follows:

feature_folds = ms.KFold(n_splits=10, shuffle = True) # cross-validation folds

selector = fs.RFECV(estimator = your_model, cv = feature_folds,
                      scoring = 'roc_auc') 

selector = selector.fit(your_features, your_labels)

selector.support_ 
$\endgroup$
1
  • $\begingroup$ Make sure you format your answer properly before posting $\endgroup$
    – Peter
    Apr 18 '20 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.