1
$\begingroup$

Let's suppose I have a dataset which has a categorical variable and the problem I am solving is a classification one.

This categorical variable var has ['A','B','C'] as the possible set of data.

What happens to a decision tree if a new category 'D' is seen only in the validation data set (meaning: absolutely new data)? Supposing the variable var is a feature used in the tree.

With the decision tree:

Does it give an error? The decision tree stops the path and returns the not-final-node's probability?

With the Logistic Regression:

The dummy variables are zero for all the categories (I suppose), and then the model runs normally?

|improve this question
$\endgroup$
  • $\begingroup$ This is presumably implementation-specific. You used the tag scikit-learn, but that (currently) requires one-hot encoding even for decision trees, so I'm not sure exactly what you want answered. Similar questions have been asked, but mostly on StackOverflow it seems: stackoverflow.com/questions/41335718/… stackoverflow.com/questions/39804733/… stackoverflow.com/questions/50630447/… $\endgroup$ – Ben Reiniger Apr 12 at 19:57
  • $\begingroup$ So, if I understand, according to the links you refered me to, the path an individual would follow would be the one indicated if all the dummy variables are zero (derived of that categorical). $\endgroup$ – Juan Esteban de la Calle Apr 12 at 20:14
  • $\begingroup$ Only if you have preprocessed your data to do that. If you one-hot encode the validation set separately, then the columns will be different than the training ones, and sklearn will give an error in either model. If you do not one-hot encode, the models won't train on the categorical 'A','B','C' in the first place. If you one-hot encode the train and validation together, then the training set sees var_D as useless and the model will never care about that dummified feature. If you modify the columns as suggested in one of those links, then your last sentence is correct. $\endgroup$ – Ben Reiniger Apr 12 at 20:20
  • $\begingroup$ Maybe split out your question about Logistic Regression to a separate question? $\endgroup$ – jonnor Apr 13 at 0:44
2
$\begingroup$

There are no categorical value support in the decision trees used in scikit-learn. Either the values are just numbers, or they have been one-hot-encoded.

One-hot encoding with an unseen input value will change the width of the feature array, which will give an error when running decision_tree.predict(X).

Without one-hot encoding the output will most likely be the same as for the value closest to it. Because the tree learns rules on form feature[2] > 3 to distinguish say a value of 3 from a value of 4 (during training). So a value of 4,5,6... etc will be treated the same way. There are no feature[2] > 4 or feature[2] < 5 rules, since there was no data for that to cause splits that could be evaluate/optimize during training.

So if you want an error also when not using one-hot-encoding, you will have to check the range of features before you pass it to the Decision Tree.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.