1
$\begingroup$

How to decide what threshold to use for removing low-variance features?

Particularly, I have 100000 features and the variances look like:

enter image description here

Could I e.g. take the average and use it to split this to ~half?

Or some other method of grouping?

$\endgroup$
1
$\begingroup$

You can do this a few ways, which I can list in ascending order of effort:

  1. Pick a value that seems ok for you and your dataset by eye-balling it then simply cut variables below the theshold from the dataset

  2. Create a function, which given a threshold, tells you how many variables would be removed, if you used that threshold. Then create a simple plot and see if there is a certain level that seems appealing (this depends on your target model once data is ready).

  3. Use some smarter functions that d a little more for you, e.g. the NearZeroVar function in the Caret package in R

There are, however, some arguments as to whether these approaches in general are optimal. Have a look at some of the discussion on this thread over at Cross-Validated. The quote from the OP in that thread if from the book of the guy who wrote the above mentioned Caret package - Max Kuhn.

The arguments against this approach say that you may be moving variables that, although they have low variance, might actually be extremely powerful in explaining your target (dependent) variable.

A final approach I can suggest goes into the realm of covariance, that is to look at the collinearity between pairs of variables. I have done this in the past and it worked out well for me. The basic algorithm would look something like this:

  1. Compute the covariance matrix across all your variables
  2. Find the pair with the highest covariance
    • correlation if you prefer that measure
    • the plan is to remove one of these two variables
  3. using your covariance matrix from step 1, compute which of these two variables in step 2 has the highest sum of covariances with the rest of the variables
  4. remove the variable with the highest sum of covariances
  5. Repeat steps 1-4 until you arrive at your desired number of variables, or a threshold is reached in terms of covariances or individual variable variances.

EDIT:

Here is the Scikit-Learn class which can do basic variance thresholding for you - there is also a short tutorial. They also present some ways to do recursive feature selection, similar in nature to my final approach outlined above.

$\endgroup$
  • $\begingroup$ Perhaps someone would have implemented the second approach as a Python function already? $\endgroup$ – mavavilj May 10 '18 at 7:31
  • $\begingroup$ Or perhaps something like sklearn has something more advanced than this? $\endgroup$ – mavavilj May 10 '18 at 7:32
  • $\begingroup$ @mavavilj - please see the added information in the edit of my answer. $\endgroup$ – n1k31t4 May 10 '18 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.