# Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

I'm still in the process of learning, so I'm sorry if this doesn't make much sense.

I'm doing Kaggle learns micro courses, and to work with missing tabular data we learned about using pipelines with imputers for numerical data and OneHotEncoding for categorical data. On the other hand, we also learned about using just pd.get_dummies which from my understanding just works as OneHotEncoding (please, correct me if I'm wrong about this).
However, using just pd.get_dummies seems to give me better results than the other method but in the other method I'm even going as far as to impute the numerical values where as with pd.get_dummies I'm not even sure what happens to missing numerical data.

So, what's going on here?

Thanks!

PS: After we use pd.get_dummies we then use "align". I read the docs but I'm still not entirely sure the purpose of doing this

X_train = pd.get_dummies(X_train)
X_valid = pd.get_dummies(X_valid)
X_test = pd.get_dummies(X_test)
X_train, X_valid = X_train.align(X_valid, join='left', axis=1)
X_train, X_test = X_train.align(X_test, join='left', axis=1)

• i'm sorry. I'm a little lost. what is your question? :) there seem to be multiple in this post. is it why is one performing better than the other? – MattR Jul 24 '19 at 18:52
• @MattR Sort of, I guess I did ask a few questions haha. But yeah I guess the main question is why did pd.get_dummies perform better than a combination of imputing numerical values and OneHotEncoding categorical values – Hello Mellow Jul 24 '19 at 19:19
• What kind of model? – Ben Reiniger Jul 24 '19 at 19:47
• @BenReiniger XGBoost on the Housing prices dataset from Kaggle – Hello Mellow Jul 24 '19 at 19:49

"(Making) dummy variables" and "one-hot encoding" are the same thing, yes. (The phrasings originate from statistics and computer science, respectively.)

pandas.get_dummies and sklearn's OneHotEncoder both implement this process, but have some differences in their (current) implementations.

Skip to the end if you're not interested in some details that might not be relevant for your underlying question.

I think there are three basic questions to ask:

1. What do we do if the train set has levels the test set doesn't?
2. What do we do if the test set has levels the train set doesn't?
3. Do we want one dummy variable for every level? [Otherwise, you get perfect multicollinearity, bad for some models.]
• Does "missing" count as a level?

(And to be more general, production sets count here as test sets, and validation sets can count as either training or test sets depending on your usage.)

In both (1) and (2), we'll have problems applying a model fitted on the train set to the test set without rectifying the differences. In (1), we just need to provide the extra columns to the test set (which will be all-zeros, but that's what we expect). In (2) we have more serious troubles: the model has no idea what to do with those new levels. The only thing to do is to find a reasonable way to convert those new levels to the "closest" training levels (which requires domain knowledge and/or a separate analysis), or failing that, just encode those levels as all zeros (essentially converting those levels to the "reference level" of the training set, i.e. none of the training levels).

pandas is more about data manipulation and aggregation, not about modelling, so doesn't really need to care about what to do with these issues. However, align gives us a way to work it out: it will conform the two sets' column names, filling in gaps with np.nan. Using the left-join as in your code takes care of (1). (Using outer-join would take care of the programmatic problem in (2), but the train set will still have a column of all nans, so the model won't actually know what to do with those levels.) Finally, get_dummies helpfully has the options drop_first and dummy_na to take care of (3).
(pandas version 0.25)

sklearn of course is all about modeling, and cares a great deal about train/test splits. OneHotEncoder then has the fit method that bakes in the fitting set's categories (later available with the attribute categories_), and the transform method uses only those categories for later use (with e.g. test sets). It has a useful option handle_unknown, that when set to 'ignore' takes care of (1). There has been discussion/work toward giving options to handle (2) [namely, providing an "other" level], as well as dealing with missing values [currently it produces an error]. It can take care of (3) with the option drop. Things are a little more difficult here too because sklearn uses numpy arrays internally instead of dataframes, so the columns don't actually get names. (It might be worth mentioning that earlier versions of sklearn was much more restrictive on the inputs to OneHotEncoder.)
(sklearn version 0.21)

Alright, finally to the base question. It may be that the slightly different options above produce different enough encoded datasets as to cause you problems in modeling. (Something else to check: the methods are dummifying the same columns, right?) It is more likely, I think, that the imputation method you've employed has hurt performance. I'm not familiar enough with the Housing Prices dataset to know, but if the missing values aren't missing at random, then imputation might misrepresent the truth. And, since XGBoost can handle missing values without imputing, you might have more similar results by skipping this step.