Pandas has a method called get_dummies() that creates a dummy encoding of a categorical variable. Scikit-learn also has a OneHotEncoder that needs to be used along with a LabelEncoder. What are the pros/cons of using each of them? Also both yield dummy encoding (k dummy variables for k levels of a categorical variable) and not one-hot encoding (k-1 dummy variables), how can one get rid of the extra category? How much of a problem does this dummy encoding create in regression models (collinearity issues - a.k.a. dummy variable trap)?
1 Answer
One advantage of get_dummies
is that it can operate on values other than integers (so you don't need the LabelEncoder
) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True
.
One advantage of scikit-learn's OneHoteEncoder
lies in the scikit-learn API. OHE gives you a transformer which you can apply to your training and test set separately if you specify the total number of categories. This doesn't work with get_dummies
,for example, if the training set misses categories present in the test set.
You can still delete categories by simply deleting columns from the resulting numpy array (e.g. using n_values_
or feature_indices_
to see which columns correspond to the same feature). Some models work regardless, for example tree-based models. Also, L1 regularization can often set redundant features to zero (see Lasso regression).
-
$\begingroup$ can you provide link with usage info for "e.g. using n_values_ or feature_indices_" I am unable to find anything useful $\endgroup$– HrvojeDec 12, 2018 at 9:06
-
$\begingroup$ They used to be attributes of the fitted OneHoteEncoder object. They have now been deprecated. All I was trying to say is you can remove a column using the column number of the feature. With
pandas
you can also useget_loc
if you only know the feature name. $\endgroup$– oW_ ♦Dec 12, 2018 at 16:07 -
$\begingroup$ I actually resolved problem by simply adding drop_first=True in pd.get_dummies(data=X, drop_first=True) as you suggested. Exploring shape of X after I saw that 3 columns were missing for 3 categorical variables. Linear regression worked fine after that. Thank you! $\endgroup$– HrvojeDec 13, 2018 at 5:44