Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables?

For categorical variables, it is easy to say that we will split them just by {yes/no} and calculate the total gini gain, but my doubt tends to be primarily with the continuous attributes. Let's say I have values for a continuous attribute like {1,2,3,4,5}. What will be my split point choices? Will they be checked at every data point like {<1,>=1......& so on till} or will the splitting point will be something like the mean of column?


Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.


Every split in a decision tree is based on a feature. If the feature is categorical, the split is done with the elements belonging to a particular class. If the feature is contiuous, the split is done with the elements higher than a threshold. At every split, the decision tree will take the best variable at that moment. This will be done according to an impurity measure with the splitted branches. And the fact that the variable used to do split is categorical or continuous is irrelevant (in fact, decision trees categorize contiuous variables by creating binary regions with the threshold).


Although, at a theoretical level, is very natural for a decision tree to handle categorical variables, most of the implementations don't do it and only accept continuous variables:

  • This answer reflects on decision trees on scikit-learn not handling categorical variables. However, one of the scikit-learn developers argues that

At the moment it cannot. However RF tends to be very robust to categorical features abusively encoded as integer features in practice.

  • This other post comments about xgboost not handling categorical variables.
  • rpart in R can handle categories passed as factors, as explained in here
  • Lightgbm and catboost can handle categories. Catboost does an "on the fly" target encoding, while lightgbm needs you to encode the categorical variable using ordinal encoding.

Here's an example of how lightgbm handles categories:

import pandas as pd
from sklearn.datasets import load_iris
from lightgbm import LGBMRegressor
from category_encoders import OrdinalEncoder

X = load_iris()['data']
y = load_iris()['target']

X = OrdinalEncoder(cols=[3]).fit_transform(X)

dt = LGBMRegressor()
dt.fit(X, y, categorical_feature=[3])

  • 2
    $\begingroup$ It really depends on algorithm. For example decision trees used in popular Python packages (scikit-learn and XGBoost) can't handle categorical data out of the box (scikit-learn for example uses CART algorithm) $\endgroup$ – Jakub Bartczuk Jun 4 '18 at 18:39
  • $\begingroup$ Yes, that was pretty much helpful @DavidMasip. I actually had confusion regarding particulary continuous variables and it got cleared now :) $\endgroup$ – Sahil Chaturvedi Jun 15 '18 at 19:42

I am not sure if most answers consider the fact that splitting categorical variables is quite complex. Consider a predictor/feature that has "q" possible values, then there are ~ $2^q$ possible splits and for each split we can compute a gini index or any other form of metric. It is conceptually easier to say that "every split is performed greedily based on metric (MSE for continuous and e.g. gini index for categorical)" but it is important to addess the fact that number of possible splits for a given feature are exponential in the number of categories. It is correct observation that CART handles it without exponential complexity, but the algorithm it uses to do so is highly non-trivial, and one should acknowledge the difficulty of the task.


It depends. Some algorithms, for example ID3 are able to handle categorical variables. Other, like CART algorithm are not.

There are two basic approaches to encode categorical data as continuous.

  • One-hot encoding
  • Mean encoding

One-hot encoding is pretty straightforward and is implemented in most software packages. The drawback is that it runs into problems if you have many categories (because the number of encoding dimensions is equal to number of categories).

Mean encoding (also sometimes called target encoding) consists of encoding categories with means of target (for example in regression if you have classes 0 and 1 then class 0 is encoded by mean of response for examples with 0 and so on). There are some answers on this site on that which provide more detail. I also encourage you to see this video if you want to get more about how it works and how you can implement it (there are several ways that to do mean encoding and each has its pros and cons).

In Python you can do mean encoding yourself (some approaches are shown in the video from the series I linked) or you can try Category Encoders from scikit-learn contrib.

  • $\begingroup$ I actually want to draw it using numerical calculations and not using scikit learn. I have edited the question. $\endgroup$ – Sahil Chaturvedi Jun 4 '18 at 18:54
  • $\begingroup$ I don't see how this changes the answer. You can just manually do one-hot or mean encoding. The gini coefficient doesn't depend on datatype, it only depends on grouping and target. $\endgroup$ – Jakub Bartczuk Jun 4 '18 at 18:56

When using Decision Trees, what the decision tree does is that for categorical attributes it uses the gini index, information gain etc. But for continuous variable, it uses a probability distribution like the Gaussian Distribution or Multinomial Distribution to discriminate. The second methodology is to convert it to categorical attributes and make rules like this: if a<100 and if a<100.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.