6
$\begingroup$

Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example? (What if I have a features ranging from 0 to 9 but there is an outlier of which value is 10000?) Whether it creates a separate leaf for that outlier sample or would it be merged with some other tree's leaves?

NULL Values : Do we need to replace the null values before building the model using Decision tree or it would be taken care automatically by the decision tree technique?

Thank you.

$\endgroup$

4 Answers 4

7
$\begingroup$

Outliers: In decision tree learning, you do splits based on a metric that depends on the proportions of the classes on the left and right leaves after the split (for instance, Giny Impurity). If there are few outliers (which should be the case: if not, you cannot use any model), then they will not be relevant to these proportions. For this reason, decision trees are robust to outliers.

Null values: You have to replace them (unless the software you use already does that for you, which is not generally the case).

Edit about outliers:

What I have said in outliers is only about classification trees. However, it is certainly not true in regression tress. Regression tree split criterium depends on the averages of the two groups that are splitted, and, as the average is severly affected by outliers, then the regression tree will suffer from outliers. There are two main approaches to solve this problem: either remove the outliers or build your own decision tree algorithm that makes splits based on the median instead of the average, as the median is not affected by outliers. However, basing the tree algorithm on the median will be very slow, as computing the median is way slower than computing the average.

$\endgroup$
0
6
$\begingroup$

Generally speaking, decision trees are able to handle outliers because their leafs are constructed under metrics which aim to discriminate as much as possible the resulting subsets. Whether you are using Gini Impurity, Information Gain or Variance Reduction to construct your decision tree does not change the outcome : all of these models aim to create as large (and homogeneous) buckets as possible. Under this approach, what matters is to understand the general behavior of your features. Individual behaviors (outliers) are disregarded because you gain very little information from them.

Assuming that your outliers represent a tiny proportion of your dataset, it is very unlikely that a leaf will be created to isolate them (at least in the first steps of creation of your decision tree), because you will gain very little information on the complimentary subset. However, if you have a large amount of outliers (but then are they really outliers ?), and that they tend to have the same outcome, chances are that you may get a leaf to isolate them.

On the other hand, null values should be treated whether it is through replacement, transformation or deletion from your observations. This would depend on your dataset.

  • If the amount of null values is quite significant in your dataset, you should consider creating an additional feature stating whether the value is missing or absent. Some implementations handle that directly by creating a boolean, or replace the missing values by an "outlier" value.
  • If the amount of null values is quite insignificant, and your dataset is large enough, you should consider deleting them, because it is the simpler and safer approach.
  • Else, you might try to replace them by an imputed value, whether it is mean, median, modal, or another value that you may calculate from your features.
$\endgroup$
1
  • $\begingroup$ Thanks @AshOfFire Information you have given is really useful. I got the clear picture now. $\endgroup$
    – deepguy
    May 9, 2018 at 14:42
4
$\begingroup$

The answers above are fantastic.

Additionally, what you could do is create a new column and label outliers as 1 (otherwise 0). This is a technique used in Kaggle competitions.Add a New Column for Outliers

The idea is to make it easier for the algorithm to detect patterns. A decision tree might detect faster a pattern if you feed him 1/0 for outliers.

How do you define outliers? Difficult question, but you could use IQR * 1.5.

$\endgroup$
1
$\begingroup$

Popular implementations of decision tree algorithms require you to replace or remove the null values, but the original C4.5 algorithm by Quinlan (father of the decision tree algorithms) specifically designed the algorithm to be able to handle missing values.

See the discussion at the following link for a plain language explanation:

https://www.quora.com/In-simple-language-how-does-C4-5-deal-with-missing-values

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.