# Decision tree where identical set of features results in different outcomes

I am following the example described in this page to test my decision tree program.

The initial data set is

age            astigmatism, tp-rate, contact-lenses
---------------------------------------------------
young,          no,  normal,  soft
young,          yes, reduced, none
young,          yes, normal,  hard
pre-presbyopic, no,  reduced, none
pre-presbyopic, no,  normal,  soft
pre-presbyopic, yes, normal,  hard
pre-presbyopic, yes, normal,  none
pre-presbyopic, yes, normal,  none
presbyopic,     no,  reduced, none
presbyopic,     no,  normal,  none
presbyopic,     yes, reduced, none
presbyopic,     yes, normal,  hard


age, astigmatism, tp-rate are features and the type of contact lenses is the outcome.

When I trace the example (where features with lowest entropies are used as the decision nodes), we reach the following situation a little further down the page:

The blue circles are obviously mine to make tracing easier. As we see in the constructed tree up to this point, the path

[Astigmatism = yes] -> [TP-Rate = normal] -> [Age = PP (pre-presbyopic)]


will lead to the outcome of [Contact-lenses = None].

But when you look at the data, we see that this path can lead to either a hard or a none type of contact lens.

So has the example made the right decision?

And if not, what should we do in situations where the split data-set ends up, under an identical set of features, with more than one outcome?

• Note this can only happen if the depth of the tree is limited/fixed. Otherwise you could just keep it growing until all the data in a given split is only of a single class. Aug 14, 2018 at 14:57

The task in hand can be modeled as a classification problem.
Meaning based on some given features we aim to predict the 'right' class of the outcome.
Once trained, most classification models, decisions tree included, will predict the most probable class given a set of input features.
Learning what is the most probable class, is usually done in the 'fitting' step of model training.
In your example the set of features results in 2 possible classes(hard, none) meaning that there is a 50% chance being either.
The classifier is just as wrong as its right in predicting the 'none' label.

** Some decisions tree implementations include an option to predict the probability for each class, and predict a class only if a specific confidence threshold is defined.

• Is the solution given in the page correct (just choosing one of the paths)? Does adding the probability for each path resolve the problem? Aug 14, 2018 at 9:39
• Yes, it will provide better clarity on the inner working of the classification. Aug 14, 2018 at 13:12

You could re-code your contact lens column into 3 new columns, each a binary indicator of one of the lens types. This would allow you to see, for your example, the probability of Yes | Normal | PP leading to none (say 75%) and in a separate tree it leading to hard (25%). This situation where any unique combination of features has multiple outcomes (given a sufficient sample size) you will probably find to be the norm, not the exception.

Since these two data points have identical features, they will always predict same output, as what machine learning algorithms learn is the mapping from input to output. That being said, they still have different output, since they are different in other features not included in your dataset.

Therefore, machine learning algorithm cannot distinguish the difference between such pair of data since their input are completely the same. Actually, such problem always exist. Generally, decision trees will predict the majority for each leaf. There are two things you can do though:

1. As DaFanat said, output a probability to show the uncertainty.
2. Include more features to reduce the uncertainty.