7
$\begingroup$

Let's say I have 3 categorical and 2 continuous attributes in a dataset. How will I build a decision tree using these 5 variables.?
Edit:
For categorical variables, it is easy to say that we will split them just by {yes/no} and calculate the total gini gain, but my doubt tends to reflect for particularly continuous attribute. Let's say if I have values for a continuous attribute like {1,2,3,4,5}. What will be my split point choices? Will they be checked at every data point like {<1,>=1......& so on till} or will the splitting point will be something like the mean of column?

$\endgroup$
9
$\begingroup$

Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.

Edit:

Every split in a decision tree is based on a feature. If the feature is categorical, the split is done with the elements belonging to a particular class. If the feature is contiuous, the split is done with the elements higher than a threshold. At every split, the decision tree will take the best variable at that moment. This will be done according to an impurity measure with the splitted branches. And the fact that the variable used to do split is categorical or continuous is irrelevant (in fact, decision trees categorize contiuous variables by creating binary regions with the threshold).

$\endgroup$
  • $\begingroup$ It really depends on algorithm. For example decision trees used in popular Python packages (scikit-learn and XGBoost) can't handle categorical data out of the box (scikit-learn for example uses CART algorithm) $\endgroup$ – Jakub Bartczuk Jun 4 '18 at 18:39
  • $\begingroup$ Cart algorithm can handle them $\endgroup$ – David Masip Jun 4 '18 at 18:41
  • $\begingroup$ Yes, that was pretty much helpful @DavidMasip. I actually had confusion regarding particulary continuous variables and it got cleared now :) $\endgroup$ – Sahil Chaturvedi Jun 15 '18 at 19:42
4
$\begingroup$

I am not sure if most answers consider the fact that splitting categorical variables is quite complex. Consider a predictor/feature that has "q" possible values, then there are ~ 2^q possible splits and for each split we can compute a gini index or any other form of metric. It is conceptually easier to say that "every split is performed greedily based on metric (MSE for continuous and e.g. gini index for categorical)" but it is important to addess the fact that number of possible splits for a given feature are exponential in the number of categories. It is correct observation that CART handles it without exponential complexity, but the algorithm it uses to do so is highly non-trivial, and one should acknowledge the difficulty of the task.

$\endgroup$
1
$\begingroup$

It depends. Some algorithms, for example ID3 are able to handle categorical variables. Other, like CART algorithm are not.

There are two basic approaches to encode categorical data as continuous.

  • One-hot encoding
  • Mean encoding

One-hot encoding is pretty straightforward and is implemented in most software packages. The drawback is that it runs into problems if you have many categories (because the number of encoding dimensions is equal to number of categories).

Mean encoding (also sometimes called target encoding) consists of encoding categories with means of target (for example in regression if you have classes 0 and 1 then class 0 is encoded by mean of response for examples with 0 and so on). There are some answers on this site on that which provide more detail. I also encourage you to see this video if you want to get more about how it works and how you can implement it (there are several ways that to do mean encoding and each has its pros and cons).

In Python you can do mean encoding yourself (some approaches are shown in the video from the series I linked) or you can try Category Encoders from scikit-learn contrib.

$\endgroup$
  • $\begingroup$ I actually want to draw it using numerical calculations and not using scikit learn. I have edited the question. $\endgroup$ – Sahil Chaturvedi Jun 4 '18 at 18:54
  • $\begingroup$ I don't see how this changes the answer. You can just manually do one-hot or mean encoding. The gini coefficient doesn't depend on datatype, it only depends on grouping and target. $\endgroup$ – Jakub Bartczuk Jun 4 '18 at 18:56
-1
$\begingroup$

When using the Decision Tree, What decision tree does is this that for categorical attributes it uses the gini index, information gain etc. But for Continuous Variable it uses a probability distribution like Gaussian Distribution or Multinomial Distribution to discriminate. Second methodology is to convert it to categorical attribute and make rules like this if a<100 and if a<100

$\endgroup$
-2
$\begingroup$

If you really know the definition of Gini Index, you wouldn't have posted this question. You should not be worried about type of variable. It's to do with homogeneity of splitted classes.

100% correctly said by @david masip. Also, Do enough research before you post. :P

$\endgroup$
  • $\begingroup$ I agree, I actually posted a very basic question, but the motive of my question was that for continuous variable particularly there is some sort of confusion as to what to choose as the splitting point. For categorical variables it is easy as {yes/no}, but in case of continuous attributes it is little vague to understand. I have edited the question for the purpose of your understanding, $\endgroup$ – Sahil Chaturvedi Jun 15 '18 at 18:49
  • $\begingroup$ datascience.stackexchange.com/questions/32129/… $\endgroup$ – Arpit Sisodia Jun 18 '18 at 6:26
  • 1
    $\begingroup$ Although one answered there. $\endgroup$ – Arpit Sisodia Jun 18 '18 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.