Ordinal feature in decision tree

Question

I am curious if ordinal features are treated differently from categorical features in decision tree, I am interested in both cases where target is categorical or continuous.

If there is a difference, could you anybody point to good source with explanation and any packages (R or Python) supporting it?

Answer 1

As per my knowledge, it doesn't matter for a decision tree model whether the features are ordinal or categorical. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. Decision trees describe patterns by using a list of attributes.

For a more detailed explanation, I am providing here some links which you will find helpful for such queries.
http://www.ibm.com/support/knowledgecenter/SS3RA7_17.0.0/clementine/nodes_treebuilding.html

http://scikit-learn.org/stable/modules/tree.html

http://www.ryerson.ca/~rmichon/mkt700/SPSS/Creating%20Decision%20Trees.htm

Answer 2

Ordinal data and Categorical data are treated exactly the same in GBM. However, the results will most likely be different if you have ordinal data formatted as categorical. This would also be true of categorical data formatted as ordinal.

Ordinal data as Categorical data

Let us consider the following example. You are predicting weights of humans. You have the mapping {'S': 0, 'XL': 1, 'L': 2, 'M': 3} for a feature of t-shirt size. The GBM does its splitting down to the last branch and has yet to split on the t-shirt size feature. At this point it tries to split on t-shirt size but finds no gain as a single split can only group ['S', 'XL'] and ['M', 'L'].

If we did encode ordinally then we would get {'S'=0, 'M'=1, 'L'=2, 'XL=3'} as our mapping. Thus the final split would result in ['S', 'M'] and ['L', 'XL'] and would certainly result in a gain.

Now I understand that this is a very contrived example and that tuning the parameters correctly would eventually split as such

        ['S', 'XL', 'L', 'M']             
           /            \                 No Gain, maybe loss (-2)
     ['S', 'XL']      ['M', 'L']
       /       \         /      \         Big gain (+15)
    ['S']   ['XL']     ['M']   ['L']

Why though, would you leave this up to tricky parameter tuning, longer training of models and potential loss of information when you can provide the GBM with known information. It makes sense to encode ordinality in this scenario.

Categorical Data as Ordinal

Let us say we now have employment status as categorical data for weight prediction. We label this as follows {'unemployed': 0, 'employed': 1, 'self_employed':2, 'education':3}. We can assume over a large dataset that the 4 differences will has little effect on weight. However, if the dataset is sufficiently large there is a good chance that there will be a branch where the ordinality appear to have some relationship to weight. The model would then learn this and split accordingly. This would cause overfitting.

In this case if we represent this data

Employment_Status 
0
2

as follows (Binary Encoding) this relationship would not be learnt as they are being considered as different features. ES stands for Employment Status.

ES_unemployed    ES_employed    ES_self_employed    ES_education
1                0              0                   0
0                0              1                   0

Again, this is a contrived example. In this case you could overcome the overfitting in conventional methods such as through cross validation but this is extremely time consuming. IMO it would be well worth doing the correct feature engineering first and treating the categorical features as categorical and ordinal as ordinal.