How to decide what threshold to use for removing low-variance features?

Question

How to decide what threshold to use for removing low-variance features?

Particularly, I have 100000 features and the variances look like:

Could I e.g. take the average and use it to split this to ~half?

Or some other method of grouping?

Answer 1

You can do this a few ways, which I can list in ascending order of effort:

1. Pick a value that seems ok for you and your dataset by eye-balling it then simply cut variables below the theshold from the dataset

2. Create a function, which given a threshold, tells you how many variables would be removed, if you used that threshold. Then create a simple plot and see if there is a certain level that seems appealing (this depends on your target model once data is ready).

3. Use some smarter functions that d a little more for you, e.g. the NearZeroVar function in the Caret package in R

There are, however, some arguments as to whether these approaches in general are optimal. Have a look at some of the discussion on this thread over at Cross-Validated. The quote from the OP in that thread if from the book of the guy who wrote the above mentioned Caret package - Max Kuhn.

The arguments against this approach say that you may be moving variables that, although they have low variance, might actually be extremely powerful in explaining your target (dependent) variable.

A final approach I can suggest goes into the realm of covariance, that is to look at the collinearity between pairs of variables. I have done this in the past and it worked out well for me. The basic algorithm would look something like this:

1. Compute the covariance matrix across all your variables
2. Find the pair with the highest covariance
   • correlation if you prefer that measure
   • the plan is to remove one of these two variables
3. using your covariance matrix from step 1, compute which of these two variables in step 2 has the highest sum of covariances with the rest of the variables
4. remove the variable with the highest sum of covariances
5. Repeat steps 1-4 until you arrive at your desired number of variables, or a threshold is reached in terms of covariances or individual variable variances.

EDIT:

Here is the Scikit-Learn class which can do basic variance thresholding for you - there is also a short tutorial. They also present some ways to do recursive feature selection, similar in nature to my final approach outlined above.