I have a dataset with a categorical variable year which has the years:: 2015,2016,2017,2018,2019. What I am trying to understand is how does a classifier work on this feature? Let's take a decision tree or XGboost classifier. It goes ahead and does the split when it is the turn for this feature but if I have a test dataset whose year column has values such as 2020 or 2021, then what happens? what would the classifier do in this case?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
13
Be careful that this 'year' feature is not categorical, it's actually numerical (ordinal at least) because the values are ordered. It's important because the data can potentially contain some patterns following this order: for example, maybe the probability to have label Y with feature X tends to decrease along the years. If there is such a pattern, the model can use this information even in a year not seen in the training data, for example it can calculate that probability of Y with feature X is even lower in 2021.
This would be different with a categorical feature: for example, imagine you have a feature with a company name like 'Apple', 'Microsoft', 'Google', 'Samsung'. There's no natural order between these names, so they would be represented with one-hot encoding, as 4 independent features. In this case the classifier cannot do anything with a new name, because it cannot even represent any pattern involving a new name.
-
$\begingroup$ Thanks for your response, understood but suppose that I have a feature like company and we have 'Apple' and 'IBM' but 'Microsoft' pops up in the test set. This can happen in reality. So what should I do by then? How should I train my model? What kind of model do I need to train? What is the right approach in such circumstances? $\endgroup$ Jul 18, 2022 at 16:39
-
$\begingroup$ @ Erwan @Brian Spiering in case of ordinal features, is the ordinal OneHot encoding of scikit learn the way to go? or there are other ways to tackle this? suppose that we have grades column, having values from A to F. Is this ordinal? if so, one hot encoder encodes them such that A: 0 and F:5 while I know A should get 5 points and F zero points. Do I need to refine the encoded values based on the requirement such that the highest rank/order gets the highest encoded value? $\endgroup$ Jul 18, 2022 at 17:36
-
$\begingroup$ @RaminSalimi for question 1: it's just impossible to take into account new values of a categorical feature, at least with regular classification. The standard way to deal with it would to periodically re-train a new model using all the available (including recently annotated) data. Question 2: if the feature is ordinal, I would suggest a numerical encoding. yes, grades as letters would be an ordinal feature. imho one-hot encoding those would be really bad. In particular, an error between A and F would be the same as an error between B and C. With ordinal encoding these would be represented ... $\endgroup$– ErwanJul 18, 2022 at 18:34
-
$\begingroup$ ... as numbers, so the error between A=0 and F=5 would be considered larger than between B=1 and C=2. The model could use this information to minimize the global error level, but with OHE it can't. $\endgroup$– ErwanJul 18, 2022 at 18:36
-
$\begingroup$ What do you mean by saying an error between A & F would be the same as the error between B & C? How do we calculate the error? so I didn't understand because the difference between A & F should be larger than the difference between B & C, right? Also how should we do numerical encoding? $\endgroup$ Jul 18, 2022 at 18:58
$\begingroup$
$\endgroup$
1
Year is better modeled as an ordinal feature, not a categorical feature.
A tree-based classifier will learn to split the data to create homogenous regions. For example, the model will learn that years less than 2017 are associated with a certain label and 2017 and greater years are associated with a different label.
Tree-based models do not extrapolate, thus will not make valid predictions for years outside of the observed range that appeared during training.
answered Jul 16, 2022 at 16:54
-
$\begingroup$ So based on your response, can we say that for cardinal features, the test set range has to be the same? or it is true only about the tree-based models? How about the categorical features, if in the test set, we observe something that did not exist amongst the categories of a feature in the train set, what happens then? $\endgroup$ Jul 17, 2022 at 6:34