How to make a decision tree with both continuous and categorical variables in the dataset?

Question

Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables?

Edit:
For categorical variables, it is easy to say that we will split them just by {yes/no} and calculate the total gini gain, but my doubt tends to be primarily with the continuous attributes. Let's say I have values for a continuous attribute like {1,2,3,4,5}. What will be my split point choices? Will they be checked at every data point like {<1,>=1......& so on till} or will the splitting point will be something like the mean of column?

Answer 1

Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.

Theory

Every split in a decision tree is based on a feature. If the feature is categorical, the split is done with the elements belonging to a particular class. If the feature is contiuous, the split is done with the elements higher than a threshold. At every split, the decision tree will take the best variable at that moment. This will be done according to an impurity measure with the splitted branches. And the fact that the variable used to do split is categorical or continuous is irrelevant (in fact, decision trees categorize contiuous variables by creating binary regions with the threshold).

Implementation

Although, at a theoretical level, is very natural for a decision tree to handle categorical variables, most of the implementations don't do it and only accept continuous variables:

• This answer reflects on decision trees on scikit-learn not handling categorical variables. However, one of the scikit-learn developers argues that

At the moment it cannot. However RF tends to be very robust to categorical features abusively encoded as integer features in practice.

• This other post comments about xgboost not handling categorical variables.
• rpart in R can handle categories passed as factors, as explained in here
• Lightgbm and catboost can handle categories. Catboost does an "on the fly" target encoding, while lightgbm needs you to encode the categorical variable using ordinal encoding.

Here's an example of how lightgbm handles categories:

import pandas as pd
from sklearn.datasets import load_iris
from lightgbm import LGBMRegressor
from category_encoders import OrdinalEncoder

X = load_iris()['data']
y = load_iris()['target']

X = OrdinalEncoder(cols=[3]).fit_transform(X)

dt = LGBMRegressor()
dt.fit(X, y, categorical_feature=[3])

Answer 2

I am not sure if most answers consider the fact that splitting categorical variables is quite complex. Consider a predictor/feature that has "q" possible values, then there are ~ $2^q$ possible splits and for each split we can compute a gini index or any other form of metric. It is conceptually easier to say that "every split is performed greedily based on metric (MSE for continuous and e.g. gini index for categorical)" but it is important to addess the fact that number of possible splits for a given feature are exponential in the number of categories. It is correct observation that CART handles it without exponential complexity, but the algorithm it uses to do so is highly non-trivial, and one should acknowledge the difficulty of the task.

Answer 3

It depends. Some algorithms, for example ID3 are able to handle categorical variables. Other, like CART algorithm are not.

There are two basic approaches to encode categorical data as continuous.

• One-hot encoding
• Mean encoding

One-hot encoding is pretty straightforward and is implemented in most software packages. The drawback is that it runs into problems if you have many categories (because the number of encoding dimensions is equal to number of categories).

Mean encoding (also sometimes called target encoding) consists of encoding categories with means of target (for example in regression if you have classes 0 and 1 then class 0 is encoded by mean of response for examples with 0 and so on). There are some answers on this site on that which provide more detail. I also encourage you to see this video if you want to get more about how it works and how you can implement it (there are several ways that to do mean encoding and each has its pros and cons).

In Python you can do mean encoding yourself (some approaches are shown in the video from the series I linked) or you can try Category Encoders from scikit-learn contrib.

Answer 4

When using Decision Trees, what the decision tree does is that for categorical attributes it uses the gini index, information gain etc. But for continuous variable, it uses a probability distribution like the Gaussian Distribution or Multinomial Distribution to discriminate. The second methodology is to convert it to categorical attributes and make rules like this: if a<100 and if a<100.