Dealing with Extra Categories in Test Set

Question

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific variable x which is a categorical variable; suppose that it takes values good and best in the training data. I would be inclined to use an ordinal encoder, such as OrdinalEncoder from sklearn.preprocessing. This would map good --> 1 and best --> 0, say.

Suppose that the model I'm using requires no NAs. My hypothetical dataset is lovely and has no NAs! Grand. I train it.

I now come to the test set. In this, the variable x sees a new value: bad. I would, obviously, have wanted to set this to 2. What should I do? Should I look at the entire dataset when encoding? This seems dodgy. Plus, if I add more data in the future, I might run into the same issue: maybe I see really bad.

Might this simply be classed as "bad practice". I should make sure that I know all the options in advance so that I can encode them appropriately in the first place.

If I were doing one-hot encoding, such as with OneHotEncoder, I'd be fine. I'd just write a 0 in the column representing "is x good/bad?" and be done. But something more intelligent needs to be done with the ordinal version. Is it ok to just stick in a value of 2 retrospectively? Seems dodgy...

Answer 1

You can reserve a special ordinal value to indicate "unknown/unseen during training." You would use this special value for any and all values of x that you encounter in the test set and in production.

In fact, scikit-learn's OrdinalEncoder does this for you via the handle_unknown parameter.

Answer 2

Regarding the case when you have 'bad' category present in the entire dataset, I would recommend using sklearn.model_selection.train_test_split function, with stratify option set to a corresponding variable. If stratify option is set to a list of all categories, it will be guaranteed that every single category will be included in both training and testing datasets, not only that, every category will be present in roughly the same proportion as in the list we assign to stratify option. Here's how you do that,

from sklearn import datasets
from sklearn.model_selection import train_test_split

iris = datasets.load_iris()
X = pd.DataFrame(iris.data)
np.random.seed(43)
X['cat'] = np.random.choice(['good','best','bad','worst'],len(X))
y = iris.target

# you need only this single line
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=43, stratify = X.cat)

sns.histplot(x=X_train.cat, stat="density", alpha=0.2, color='gold', label='train');
sns.histplot(x=X_test.cat, stat="density", alpha=0.2, color='green', label='test');
sns.histplot(x=X.cat, stat="density", alpha=0.2, color='coral', label='entire');
plt.legend();

In the second case, when your entire dataset does not contain 'bad' category, there are different approaches how to handle that, but in my opinion, the best thing would be to retrain your model using this new category.

Answer 3

You have to remember your training data has to have all the possible values you know you will have in the test set. The data set should be shuffled before splitting so your case should not append. Remember a model cannot predict correctly on unknown category value never seen during training. So always shuffle and/or get more data so every category values are included in the data set.