Different number of features after using OneHotEncoder

I have train and test data in two separate files.

OneHotEncoder gives different number of features for Train and Test Data based on the different values they have. But the classifier requires that the number of features for test and train data should be equal, how can I solve this problem?

• Create features (one hot encoding) before splitting into train and test sets. – Ankit Seth Mar 13 '18 at 12:55
• Yes. You have to make sure you take all data to one hot encoding so that you get all features. I would suggest that you create a dummy variable for train and test data so that you can use it to split the combined data after your encoding process. – Ankit Seth Mar 13 '18 at 13:04
• It‘s a red flag that one feature expression only occurs in train/test but not both. Make sure to double check if it even makes sense to include the feature in the model at all! – AlexR Mar 13 '18 at 13:25
• If you categorical features represent high cardinality (and of course distinct in test and train), OneHotEncoder is not a way-to-go method, besides other issues it may cause! Everyone here is talking about combining train+test and do encoding, but that is usually a wrong practice. Test is just a sample of unseen data, which can have different subcategories of a particular feature! In machine learning, we just do not want to build a model for one time use. It is expected to predict on incoming flow of unseen data, otherwise why bother! Combining train and test is against this principle. – TwinPenguins Mar 14 '18 at 22:07
• @Sameed Well, I am not 100% sure what are the best practices. The more I study the more I learn that OneHotEncoder is not the only one and often not the best. I suggest you search a bit more. Maybe you wanna check Catboost (tech.yandex.com/catboost/doc/dg/concepts/…) and the ways they offer to do categorical encoding beyond OneHotEncoder for Gradient Boosting Trees, or a Python implementation for various categorical encoding: github.com/scikit-learn-contrib/categorical-encoding! – TwinPenguins Mar 16 '18 at 8:56

One hot encoding is only a symptom. The cause of the problem is that your factor valible has not the same leveles in the test and train data.

Here you should distinct. Is it only a problem of sampling? You created your test data as (say) 20% sample of the original data. Some levels with small cardinalty could fail to get in the sample. If it is the case you must take care to sample all levels and get 20% of data for each level.

Other problem is if your factor valible is not static and through the time new lavels can emerge. Here is it realy possible to encounter new levels in "unseen data".

One possible approach to handle this is to train an explicit unknown level based on some prepared average data. In the preprocessing phase all new levels are recognised and mapped to this unknown level.

Periodlcally refresh the model to include the recent appeared levels.

One hot encoding is a way of converting output label for 3 categories like 2 into [0, 1, 0] or 3 into [0, 0, 1].

If you are using scikit learn to convert the value into one hot encoder then in training time you should use

enc = OneHotEncoder()
enc.fit(x_train)


If you are using scikit learn to convert the value into one hot encoder then in testing time you should use enc.transform(x_test)

The reason we are using transform function in case of testing is we have to consider the label values on the basis of which we have converted data in training time. Because in testing time we not get all labels for that column

Data preprocessing (including creation of dummy variables from categorical features) needs to be done before splitting the data into train and test set. This would solve your issue.

You don't have much details on what you are trying to do - so if what I say is irrelevant just skip it; but, the fact that your test set contains less categories than the training set is something that I would try to avoid. If this happens for multiple categories, maybe you should try to group some of the categories together into more general ones (e.g. if the variable contains "ways of going to work", you could merge "bus", "tram", "metro" into "public transport"). Also, why don't you try cross-validation instead of having a static test set? (Useful link for cross-validation with scikit-learn)