1
$\begingroup$

I'm working on an unsupervised clustering problem. I read multiple times that a variable with higher variance can be chosen over a variable with a lower variance. For example, scikit-learn implements a function that removes features with a variance lower than a threshold. (sklearn.feature_selection.VarianceThreshold)

However, isn't the variance entirely dependent on scale/measurement unit? If I standardize my features, the variance is 1 for all of them.

$\endgroup$

2 Answers 2

1
$\begingroup$

You are correct that variance depends on the scale and typically it is not a good proxy for how informative a feature will be in terms of the response. The exception is zero variance features. A feature with zero variance has the exact same value for all observations and is therefore completely uninformative. Sklearns VarianceThreshold function defaults to removing only the features with exactly zero variance.

Another group of non-informative features is the near-zero-variance feature. However, typically, classifying a feature as a near-zero-variance feature does not actually rely on computing the feature's variance at all. See, for example, the NearZeroVar function documentation from R's caret package:

"nearZeroVar diagnoses predictors that have one unique value (i.e. are zero variance predictors) or predictors that have both of the following characteristics: they have very few unique values relative to the number of samples and the ratio of the frequency of the most common value to the frequency of the second most common value is large."

You can see that the algorithm is not scale-dependent and does not rely on variance.

$\endgroup$
0
$\begingroup$

If a variable is constant (zero variance), then it wouldn't help in predictive modeling. Similarly, if a variable doesn't change much, it would not help much as well.

Since removing noisy variables are better than keeping them, removing such variables can improve the modeling.

If sklearn.feature_selection.VarianceThreshold relies only on the variance of the features, then it will not be effective on standardized variables.

See some info here https://scikit-learn.org/stable/modules/feature_selection.html#variance-threshold

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.