3
$\begingroup$

Pandas has a method called get_dummies() that creates a dummy encoding of a categorical variable. Scikit-learn also has a OneHotEncoder that needs to be used along with a LabelEncoder. What are the pros/cons of using each of them? Also both yield dummy encoding (k dummy variables for k levels of a categorical variable) and not one-hot encoding (k-1 dummy variables), how can one get rid of the extra category? How much of a problem does this dummy encoding create in regression models (collinearity issues - a.k.a. dummy variable trap)?

$\endgroup$
7
$\begingroup$

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True.

One advantage of scikit-learn's OneHoteEncoder lies in the scikit-learn API. OHE gives you a transformer which you can apply to your training and test set separately if you specify the total number of categories. This doesn't work with get_dummies ,for example, if the training set misses categories present in the test set.

You can still delete categories by simply deleting columns from the resulting numpy array (e.g. using n_values_ or feature_indices_ to see which columns correspond to the same feature). Some models work regardless, for example tree-based models. Also, L1 regularization can often set redundant features to zero (see Lasso regression).

$\endgroup$
  • $\begingroup$ can you provide link with usage info for "e.g. using n_values_ or feature_indices_" I am unable to find anything useful $\endgroup$ – Harvey Dec 12 '18 at 9:06
  • $\begingroup$ They used to be attributes of the fitted OneHoteEncoder object. They have now been deprecated. All I was trying to say is you can remove a column using the column number of the feature. With pandas you can also use get_loc if you only know the feature name. $\endgroup$ – oW_ Dec 12 '18 at 16:07
  • $\begingroup$ I actually resolved problem by simply adding drop_first=True in pd.get_dummies(data=X, drop_first=True) as you suggested. Exploring shape of X after I saw that 3 columns were missing for 3 categorical variables. Linear regression worked fine after that. Thank you! $\endgroup$ – Harvey Dec 13 '18 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.