Should columns with close to zero variance be removed before or after one hot encoding?

Question

I've been using R and the caret package since a while. The caret package provides a function to reduce the number of variables - nearZeroVar() wherein variables that have close to zero variance are returned and one can remove them from the dataframe

Caret also provides another function dummyVars that converts factors to dummy variables.

Now my question is - what should the order of using the two be? Should the features with near zero variance be thrown out and then converted to dummies or should it be the other way around? I have a feeling that the order matters because in some cases dummy variables might have a lot of zeros but still be important.

Answer 1

I think 1st you need to remove the columns with near to zero variance(assuming that they are not nominal variables), only if you have more variables as the computation time can be reduced.

Let us consider a scenario where there are 2 columns col1, col2 and you checked for variance then col1 had 0.003 variance and col2 had 0.5 var. once you apply the function nearZeroVar(), it removes col1, you need to hot-encode only 1 variable.

By that you can save time and your computer performance won't be effected.

Answer 2

I'd plot your data and see if the proportion of the label you are trying to predict is similar or different among the labels for your feature in question. If similar, you can probably safely remove the column using the nearZeroVar() function. If different - then the feature is predictive! Keep it, and convert the column into dummy variables.

For example, imagine I was trying to use a diamond's color (a categorical variable) to determine whether it was expensive (a categorical variable I invent below...)

require(dplyr)
require(ggplot2)
data('diamonds')

diamonds %>% 
  mutate(expensive = price > 5000) %>%
  group_by(color,expensive) %>%
  summarise(n = n()) %>%
  mutate(freq = n/sum(n)) %>% 
  filter(expensive) %>%
  ggplot() + 
  geom_bar(aes(color,freq),stat='identity') + 
  labs(x='color',y='proportion expensive')

And you get a plot like this -

where there is a lot of variance in the target label proportion among your feature labels (even if one is pretty rare), I'd keep it as a predictor. If instead the proportion is very similar - you might as well drop it.

Answer 3

Your feeling is right, in some cases the order might matter.

Applying nzv after dummyVars means that you lump together very rare levels of nominal predictors. As Tom suggested, you have to make this decision depending on the properties of your particular dataset.

Some examples:

1. You have a classification task and want to identify fraudulent sellers (very imbalanced dataset!). If you apply nzv after dummyVars to the predictor variable „accepted payment type“ you’d lump the level „accept only western union transfers“ (which might be a very good indicator that such sellers are not trustworthy) with some other rare levels. This will significantly worsen the performance of your classifier.

2. In case of regression tasks, especially when your target function is smooth and nice, it is often less critical if you lump together very rare levels.

You also might want to experiment with different parameters of nzv, such as cutoff frequency.