1
$\begingroup$

Let's say I do a Decision Tree analysis.

But the performance characteristics are nothing great (e.g. ROC is nothing great).

Is there anything I can do with this "not so great" tree. Or do I typically need to trash it and try something else (either a new data set or a new analysis on the same data)?

$\endgroup$
  • 1
    $\begingroup$ If you tell us the use case for the model then the community can suggest ways in which it can still be useful. $\endgroup$ – wabbit Mar 12 '16 at 5:30
  • $\begingroup$ @Wabbit It's a very general question. I've run a few Decision Trees, and in each case, the ROC has never been as great as I see in tutorials. The last one was to do with the Stack Overflow Developer Survey data. I was trying to predict "Job Satisfaction" using Age, Experience, Compensation, Employment Status and Remote Status (i.e. telecommuting). $\endgroup$ – thanks_in_advance Mar 14 '16 at 16:37
  • 1
    $\begingroup$ I'm not sure if this is really a problem with decision trees. Decision trees have good "capacity" in terms of being able to model complex interactions between features etc. What I mean is that without doing further feature engineering a linear model might have given you even worse ROC. You can work on the following to see if your AUC improves: 1. Adding more features 2. Features transformations $\endgroup$ – wabbit Mar 24 '16 at 14:11
4
$\begingroup$

Decision trees has one big quality and one big drawback. Their big quality is that they are what is known as glass-box models. What I mean by that is that they expose what they learn in a very clear and intuitive way. The name comes from the fact that you can see through a glass box. So, because of that, decision trees are very useful for analysis, they are a nice support to understand the relations between variables, which variables are important, in which way they are important. And even if they does not provide crystal clear information, they can bring you ideas about that. This might be very helpful especially if you have domain expert knowledge and you can put things together in a meaningful manner.

Their main drawback is their high variability. This problem is mainly caused by their greedy approach. Each decision in the first level nodes shape differently the tree. You can even go further and see that a single additional data point is enough in many cases to get a totally different tree, especially if the sample is small or if the data is noisy.

There are two types of approaches to solve this issue. The first type of approach tries to improve the single tree you built. This kind of approaches are known as pruning. A simple example would be reduced error pruning. This approach is simple and produce good results. You train your tree on a sample of data. Then you take another sample of data, you fit the new data on the tree, and then evaluate again the nodes of the tree from the perspective of the new data. If a non-leaf node get at least the same error if would not be split than if it would be split, then you can decide to cut the child nodes and transform that node into a leaf node. There are however much nicer pruning strategies, which are stronger, perhaps based on cross validation or some other criteria, mostly statistical based. Notice however that for reduced error pruning you need additional data, or to split your original sample in two, one for training, the other for pruning. If you go further to estimate the prediction error you need a third sample.

The second approach would be either to build multiple times some trees and chose some based on cross validation, bootstrapping or whatever method you you, or use a tree ensembles like bagging or boosting algorithms. Note that for boosting and bagging you loose glass box property.

Ultimately you have to choose between understanding and performance, having as a decent compromise the pruning procedure.

$\endgroup$
2
$\begingroup$

Two things can be done.

  1. One is apply ensemble methods like random forest, Adaboost, etc..
  2. Second one is try to optimise your decision tree and find the best parameters using techniques like GridSearchCV.
$\endgroup$
1
$\begingroup$

Depending on the software / language you are using, you should be able to implement some form of:

  • Boosting - Wikipedia - (AdaBoost) - Reweights incorrect predictions and re-evaluates the tree
  • Bagging - Wikipedia - builds multiple decision trees by repeatedly resampling training data with replacement, and voting the trees for a consensus prediction.
  • Pruning - Wikipedia - Decision tree algorithms can create overly complex trees with too many branches and nodes that do not generalize well. This is known as overfitting. Pruning removes the extra nodes that do not provide any valuable information.
$\endgroup$
  • 1
    $\begingroup$ Welcome to DataScience SE. Can you pl elaborate your answer? Currently, it is just throwing in options instead of explaining what to use when and why? $\endgroup$ – Dawny33 Mar 11 '16 at 5:52
1
$\begingroup$

To be short: yes, trash it. Any ML model you use has one job - to predict. If it can't do that job then why on earth would you ever use it? The whole impetus for using ML algorithms is to predict better than guessing randomly. Ideally, much, much better than guessing randomly.

Understanding why a model makes a decision has some value, sure, but only insofar as that decision has value to the end user/solves a business problem.

$\endgroup$
  • $\begingroup$ A follow-up question: I was thinking that in situations where there might be a huge number of predictor variables, one of the uses of Data Mining could be to find a short-list of predictors to then do regressions (or some such) with. What's your view on doing something like that? Legit or non-legit? $\endgroup$ – thanks_in_advance Mar 14 '16 at 16:32
  • 1
    $\begingroup$ Very legit. There's a whole class of algorithms for "dimensionality reduction" and it sounds like that's what you're looking for. These algos help you take your "too many predictors" and turn them into "just enough predictors" by isolating the ones that contain the most signal to predictions. Give principle component analysis a look. $\endgroup$ – Derek Janni Mar 14 '16 at 16:37
  • $\begingroup$ I'm familiar with traditional principal components analysis. Is there a flavor of that for Decision Tree analysis too? You mentioned the importance of prediction. When using Decision Trees to essentially shortlist the most important predictors, does the ROC become less important? $\endgroup$ – thanks_in_advance Mar 14 '16 at 16:41
  • 1
    $\begingroup$ Decisions Trees do not do what you want them to in that regard - by default the algorithm considers every feature you give it, even if there's not a lot of signal it will still be used. The only way to get a "shortlist" is with a technique that explicitly reduces the dimensionality of the feature space. $\endgroup$ – Derek Janni Mar 14 '16 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.