# Variance threshold with python problem

i’am a beginner in scikit-learn and i’ve a little problem when using feature selection module VarianceThreshold, the problem is when i set the variance Var[X]=.8*(1-.8)

it is supposed to remove all features (that have the same value in all samples) which have the probability $p>0.8$. in my case the fifth column should be removed, p=8/10>(threshold=0,7).

#####################################

from sklearn.feature_selection import VarianceThreshold
X=[[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.01,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,255,1.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00],
[0,1,2,1,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0.00,0.00,0.00,0.00,0.50,1.00,0.00,10,3,0.30,0.30,0.30,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,253,0.99,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,1,1,1,105,146,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0.00,0.00,0.00,0.00,1.00,0.00,0.00,255,254,1.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00],
[0,2,3,1,223,185,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,4,4,0.00,0.00,0.00,0.00,1.00,0.00,0.00,71,255,1.00,0.00,0.01,0.01,0.00,0.00,0.00,0.00]]
sel=VarianceThreshold(threshold=(.7*(1-.7)))


and this is what i get when running the script

>>> sel.fit_transform(X)

array([[ 1., 105., 146., 1., 1., 255., 254.],
[ 1., 105., 146., 1., 1., 255., 254.],
[ 1., 105., 146., 1., 1., 255., 254.],
[ 1., 105., 146., 2., 2., 255., 254.],
[ 1., 105., 146., 2., 2., 255., 254.],
[ 1., 105., 146., 2., 2., 255., 255.],
[ 2., 29., 0., 2., 1., 10., 3.],
[ 1., 105., 146., 1., 1., 255., 253.],
[ 1., 105., 146., 2., 2., 255., 254.],
[ 3., 223., 185., 4., 4., 71., 255.]])


the second column here should not apear. thanks;)

• This logic only works for Bernoulli distributed features (ones and zeros, where probability for a 1 is p, and then the variance is given as p*(1-p)). Your fifth/sixth column is clearly not Bernoulli distributed. – oW_ Jun 13 '17 at 15:34

0.8 is the threshold value meaning you want to exclude columns(features) with at least 80% of values is 0 or 1: sort of unbalanced dummies. This is only applied well to dummies, but not necessarily general numerical values. For numerical, you can test RFECV function in feature_selection package from sciki-learn, the syntax as follows:

feature_folds = ms.KFold(n_splits=10, shuffle = True) # cross-validation folds

selector = fs.RFECV(estimator = your_model, cv = feature_folds,
scoring = 'roc_auc')

selector = selector.fit(your_features, your_labels)

selector.support_

• Make sure you format your answer properly before posting – Peter Apr 18 '20 at 10:53