0
$\begingroup$

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific variable x which is a categorical variable; suppose that it takes values good and best in the training data. I would be inclined to use an ordinal encoder, such as OrdinalEncoder from sklearn.preprocessing. This would map good --> 1 and best --> 0, say.

Suppose that the model I'm using requires no NAs. My hypothetical dataset is lovely and has no NAs! Grand. I train it.

I now come to the test set. In this, the variable x sees a new value: bad. I would, obviously, have wanted to set this to 2. What should I do? Should I look at the entire dataset when encoding? This seems dodgy. Plus, if I add more data in the future, I might run into the same issue: maybe I see really bad.

Might this simply be classed as "bad practice". I should make sure that I know all the options in advance so that I can encode them appropriately in the first place.

If I were doing one-hot encoding, such as with OneHotEncoder, I'd be fine. I'd just write a 0 in the column representing "is x good/bad?" and be done. But something more intelligent needs to be done with the ordinal version. Is it ok to just stick in a value of 2 retrospectively? Seems dodgy...

$\endgroup$
2
  • $\begingroup$ "... and independent variables X" - if X were independent variables then any modelling would be useless. You probably meant to say that X are regressors, predictors, explanatory variables or any other synonymous term. Let's phase out misnomers. $\endgroup$ Aug 24, 2022 at 2:02
  • $\begingroup$ @paperskilltrees Statistics isn't my forte, but my understanding is that y is the dependent variable and X form the independent ones. I know people in data science like to use stats terms and then just give them new names, so maybe this is one of those. I'm not sure. My ds is knowledge is limited! $\endgroup$
    – Sam OT
    Aug 25, 2022 at 14:32

3 Answers 3

2
$\begingroup$

You can reserve a special ordinal value to indicate "unknown/unseen during training." You would use this special value for any and all values of x that you encounter in the test set and in production.

In fact, scikit-learn's OrdinalEncoder does this for you via the handle_unknown parameter.

$\endgroup$
7
  • 1
    $\begingroup$ The whole idea of ordinal encoding---as far as I understand---is that you can to, well, add order. You can then say that one value is 'closer' to one than another. Eg, 'good' is closer to 'very good' than to 'excellent'. How should I set the unknown parameter? $\endgroup$
    – Sam OT
    Sep 22, 2021 at 8:26
  • $\begingroup$ It seems to me that the right answer is instead to make sure that I know all the categories that could be chosen in advance---whether or not those are included in the training set---and encode with this. If I see something else in future data, I should view it somewhat like missing data. Although, if I have good and best as options, maybe I replace bad with good, ie the worst of the available options, rather than just make it NA. $\endgroup$
    – Sam OT
    Sep 22, 2021 at 8:26
  • $\begingroup$ Yes: if you know all possible categories beforehand, you can (and should) give each one a meaningful ordinal level. scikit-learn's OrdinalEncoder supports such a thing via the categories parameter. But if you truly do not know all the levels beforehand, or if you want your model to be robust at test time, then you'll have no choice but to treat unknown levels as unknown. $\endgroup$
    – stepthom
    Sep 22, 2021 at 10:40
  • $\begingroup$ I agree completely. But that still leaves open my question above: "How should I set the unknown parameter?" $\endgroup$
    – Sam OT
    Sep 22, 2021 at 10:56
  • $\begingroup$ You train data set should have all the category values from the start. A model can only train properly on know training examples. So shuffle data before split and get more data if needed. $\endgroup$
    – Malo
    Sep 25, 2021 at 17:17
2
$\begingroup$

Regarding the case when you have 'bad' category present in the entire dataset, I would recommend using sklearn.model_selection.train_test_split function, with stratify option set to a corresponding variable. If stratify option is set to a list of all categories, it will be guaranteed that every single category will be included in both training and testing datasets, not only that, every category will be present in roughly the same proportion as in the list we assign to stratify option. Here's how you do that,

from sklearn import datasets
from sklearn.model_selection import train_test_split

iris = datasets.load_iris()
X = pd.DataFrame(iris.data)
np.random.seed(43)
X['cat'] = np.random.choice(['good','best','bad','worst'],len(X))
y = iris.target

# you need only this single line
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
                                                    random_state=43, stratify = X.cat)

sns.histplot(x=X_train.cat, stat="density", alpha=0.2, color='gold', label='train');
sns.histplot(x=X_test.cat, stat="density", alpha=0.2, color='green', label='test');
sns.histplot(x=X.cat, stat="density", alpha=0.2, color='coral', label='entire');
plt.legend();

enter image description here

In the second case, when your entire dataset does not contain 'bad' category, there are different approaches how to handle that, but in my opinion, the best thing would be to retrain your model using this new category.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks for your suggestion, but unfortunately this doesn't answer the question at all---which is a shame, as otherwise your answer is truly excellent: very clear, with simple code and a nice figure! I have asked, "What do I do in [this] situation?" The response, "Don't be in [that] situation" doesn't answer the question, unfortunately. It is not always the way in life that one has a single large dataset and does the train--test split oneself. Alternatively, I may want to predict something with data gathered after I trained and deployed my model? Your answer doesn't handle this. $\endgroup$
    – Sam OT
    Sep 22, 2021 at 8:22
  • $\begingroup$ It seems to me that the right answer is instead to make sure that I know all the categories that could be chosen in advance---whether or not those are included in the training set---and encode with this. If I see something else in future data, I should view it somewhat like missing data. Although, if I have good and best as options, maybe I replace bad with good, ie the worst of the available options, rather than just make it NA. $\endgroup$
    – Sam OT
    Sep 22, 2021 at 8:26
  • $\begingroup$ @SamOT, thank you for the appreciation. I hope you will get the right answer soon. $\endgroup$ Sep 22, 2021 at 11:24
0
$\begingroup$

You have to remember your training data has to have all the possible values you know you will have in the test set. The data set should be shuffled before splitting so your case should not append. Remember a model cannot predict correctly on unknown category value never seen during training. So always shuffle and/or get more data so every category values are included in the data set.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks for your answer :) -- again, the "shuffle and split... should not append" doesn't apply here. What happens if you are training a model online? Maybe people are filling in a survey and one of the questions gets an extra option $\endgroup$
    – Sam OT
    Sep 27, 2021 at 8:10
  • $\begingroup$ Your training data must have all the possible categorical values to have a good working model. There is no magic. If you do not have that means your datatest is too small and you have to get more data. If you have free text fields, this is a different case, see NLP methods. $\endgroup$
    – Malo
    Sep 27, 2021 at 8:39
  • $\begingroup$ I agree completely! One should have this. I'm asking what to do if one does not have this. The response "you should have this" isn't a suitable answer, unfortunately $\endgroup$
    – Sam OT
    Sep 27, 2021 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.