6
$\begingroup$

I have a dataset on which one of the features has a lot of different categorical values. Trying to use a LabelEncoder, OrdinalEncoder or a OneHotEncoder results in an error, since when splitting the data, the test set ends up having some values that are not present in the train set.

My question is: if I choose to encode my variables before splitting the data, does this cause data leakage?

I'm aware that I shouldn't perform any normalization or educated transformations on the data before splitting the dataset, but I couldn't find a solution for this problem inside scikit-learn.

Thanks in advance for any responses.

Edit: This particular features has very high cardinality, with around 60k possible values. So using scikit-learn's OneHotEncoder with handle_unknown set to ignore would introduce too many new columns to the dataset.

| improve this question | |
$\endgroup$
  • 1
    $\begingroup$ If a not-ordered categorical feature can have 60k possible values, the best first step is probably to perform some sort of feature engineering / transformation on that to decrease that number (by e.g. grouping values or splitting the feature into a few other features). Trying to directly model the effect of those 60k categories on your target variable would require a ton of training data and would probably duplicate effort. Think of it this way: 60k rows of data means roughly 1 row for every category, and you need a lot more than 1 data point to start drawing meaningful conclusions. $\endgroup$ – NotThatGuy Jul 23 at 8:49
7
$\begingroup$

The cleanest solution would be to apply scikit's OneHotEncoder with the handle_unknown parameter set to "ignore":

handle_unknown{‘error’, ‘ignore’}, default=’error’

Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.

Other manual solution are described in this and this question on Stackoverflow, for example.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Sadly this feature has really high cardinality, with around 60k possible values. This alone made OneHotEncoder impractical. Other solutions are made to work around the limitation of the scikit library. But since this appears to be the usual approach, I'll accept this answer. Thanks, Sammy. $\endgroup$ – kaylani2 Jul 22 at 20:16
6
$\begingroup$

Encoding labels before splitting the data set should not cause leakage, particularly in the case of ordinal encoding. Ordinal encoding is just a transform from "label space" to "integer space". Changing the names we use for the labels does not add any useful information that could change classification results, so no data leakage.

Think about it this way: Suppose you have 3 labels "Red", "Blue", "Green". But, for some reason, the software package you are using only works in Spanish. So you change the labels to "Rojo", "Azul", and "Verde". No data leakage has occurred - you've just started calling the labels something different. This is almost perfectly analogous to ordinal encoding*.

I think you could make an argument that one-hot encoding allows for some very, very minor leakage. Suppose you have labels "Red", "Blue", "Green" but only the first two appear in your training set. By one-hot encoding the labels before splitting, you implicitly declare that there are three possible labels instead of two. Depending on the definition, this could be described as data leakage, since you can derive some information that's not actually included in the training set. However, I can't imagine how an ML algorithm would gain an artificial benefit in this scenario, so I don't think it's anything to worry about.

*if you ignore the fact that some algorithms can find spurious relationships between numbers, but not string labels.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ By the way, it's somewhat concerning that your test set has examples with a label not seen during training. My answer doesn't address this problem - it focuses on the data leakage question. @Sammy's suggestion is good in practice. $\endgroup$ – zachdj Jul 22 at 19:21
  • 1
    $\begingroup$ This minor information was exactly at the root of my inquiry. I'm glad to know for sure that mapping the values do not cause leakage. Regarding the unseen features in the test set, it's a problem of network attacks classification, and unknown parameters in "production" is more the norm. Thanks, zachdj. $\endgroup$ – kaylani2 Jul 22 at 20:19
  • $\begingroup$ Dummy variables / one-hot is the classic solution, but maybe you could share some info about this 60k+ category variable, it could be another solution is available. $\endgroup$ – C8H10N4O2 Jul 22 at 22:19
0
$\begingroup$

First, no data leakage here because you are encoding a feature not the target variable. Second ly, you can consider other useful encoding scheme like target encoding, which will not create a huge amount of columns like onehot encoding. In fact it creates just a single column. Also try to reduce your number of values in your category, 60k is way too many.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.