Validation Curve Interpretations for Decision Tree

Question

I'm working on a machine learning class, and we're using supervised learning right now, starting with decision trees. I'm using the UCI Credit Card dataset (whether or not certain people will default in their payments due to past history).

Using a decision tree classifier for this attempt. Running a validation curve using scikit-learn, I'm getting a plot I'm not quite sure how to interpret. As you can see from the axes, the parameter is min_samples_leaf, and I'm varying it from 1 to 30 (by 2).

Based on this plot and some Googling, I believe the correct way to interpret this is that this dataset has high bias with no variance and nothing is really being learned. Or, in other words, decision trees are not a good algorithm for this dataset, since there doesn't really seem to be a trade-off.

For max depth, I'm getting a validation curve that looks like this:

Based on what I see here, there is quite a bit of bias at the smaller set, and more variance as the depth increases. Given that GridSearchCV returns an ideal max_depth of 5 and min_samples_leaf of 19. (edit: corrected numbers). Those numbers seem to indicate a very high bias, and there really is nothing to be learned here using decision trees.

Overall, based on the min_samples_leaf, I would hesitate to recommend a decision tree for this data set. However, the learning curve and the max_depth validation curve both seem to show there might be some value.

Puzzling to me is that the accuracy score (using metrics.accuracy_score and the ideal parameters from GridSearchCV) is 82%, which doesn't seem that bad. How do I reconcile these crappy validation curves with the accuracy score?

Answer 1

A few comments on the top of my head:

• both parameters min_samples_leaf and max_depth are not very important for decision trees, so it's not surprising not to see much variation (or not all) across different values:
  • the fact that min_sample_leaf doesn't influence the performance simply means that the algorithm finds enough good predictors in the features to create leaves with a high number of instances, apparently always more than 30.
  • I'm less sure about max_depth, but I assume that it's a parameter which is used to prune the tree in order to avoid overfitting. This is exactly what happens here as soon as it's increased above 5: the algorithm creates a deeper (more complex) tree by using very specific conditions on the features which turn out to be specific to the training set, hence the divergence between training score and CV score (that's a clear sign of overfitting).
• Value 5 for max_depth is indeed optimal, as seen on the second graph (due to overfitting with higher values). The value 19 returned by grid search for min_samples_leaf is as optimal as any other value between 1 and 30, as seen on the first graph: grid search just happened to pick this one but it doesn't have any impact anyway.
• The 82% accuracy is completely normal: that's the Y axis on both graphs ;) It indeed looks like a decent performance, but we can't say for sure since there's no comparison: maybe the dataset is super easy and 82% is just the majority class, or maybe it's super hard and 82% is a great achievement.

Actually these graphs don't show much about the learning, they just show that the parameters studied are not really relevant. To observe something more interesting try:

• Ablation study: pick a random subset of the training set instances and train only with this subset. Do this for say 10%, 20%, ..., 100% of the data, then plot the performance as a function of the size of the training set. This will show how many instances are needed for the model to reach its max performance (educated guess: not that many).
• Feature selection: use only the N most informative features (as measured by information gain for instance) and plot the performance for different values of N. If you're lucky you might see an increase until a particular optimal value of N followed by a small decrease when uninformative features are added. If it happens, that last part would be due to overfitting.

Answer 2

One thing occurs to me that you are plotting the validation curve by tuning only one hyperparameter, but in the GridSearchCV function you are actually turning both hyperparameter together. in Validation_curve function you may want to turn the two parameter together and make a 3D validation plot to see a good explanation on your high matrix accuracy score.