I'm working on a machine learning class, and we're using supervised learning right now, starting with decision trees. I'm using the UCI Credit Card dataset (whether or not certain people will default in their payments due to past history).
Using a decision tree classifier for this attempt. Running a validation curve using scikit-learn, I'm getting a plot I'm not quite sure how to interpret. As you can see from the axes, the parameter is min_samples_leaf, and I'm varying it from 1 to 30 (by 2).
Based on this plot and some Googling, I believe the correct way to interpret this is that this dataset has high bias with no variance and nothing is really being learned. Or, in other words, decision trees are not a good algorithm for this dataset, since there doesn't really seem to be a trade-off.
For max depth, I'm getting a validation curve that looks like this:
Based on what I see here, there is quite a bit of bias at the smaller set, and more variance as the depth increases. Given that
GridSearchCV returns an ideal
max_depth of 5 and
min_samples_leaf of 19. (edit: corrected numbers). Those numbers seem to indicate a very high bias, and there really is nothing to be learned here using decision trees.
Overall, based on the
min_samples_leaf, I would hesitate to recommend a decision tree for this data set. However, the learning curve and the
max_depth validation curve both seem to show there might be some value.
Puzzling to me is that the accuracy score (using
metrics.accuracy_score and the ideal parameters from
GridSearchCV) is 82%, which doesn't seem that bad. How do I reconcile these crappy validation curves with the accuracy score?