2
$\begingroup$

I'm developing a binary classification tree and having some touble interpreting my training/validation curves. I used the CART algorithm with information gain as my splitting criterion. The training and test data was split 75% : 25%. There was an imbalance between my two classes, so i performed stratified (proportionate) sampling during the train/test split. Afterwards, I oversampled the minority class in my training set.

Below, i'm increasing the depth of the tree and observing the performance in terms of F1 score (harmonic mean of precision and recall; higher is better).

enter image description here

as the complexity of tree increases, my training curve approaches zero error (as expected) while my validation curve plateaus. I would expect the validation curve to decrease as the tree overfits, but here the validation curve seems to peak just as the tree is maximally overfitting the training data. Is this an indication of something I'm doing wrong, or is there a reasonable interpretation for this?

UPDATE:

I refit the tree, this time without oversampling minority class. I now have training/validation curves that exhibit more typical behavior.

enter image description here

the validation curve peaks before the tree overfits, and max performance is the same as before (~0.84), only this time with a tree that has half the complexity (depth of 14 vs 25). I am starting to think oversampling is not actually helping the tree learn.

$\endgroup$

1 Answer 1

0
$\begingroup$

You are correct that you are overfitting (i.e., a large drop in performance between training and hold-out dataset performance).

Given that decision tree algorithms are high variance, you are looking too closely at hold-out dataset performance. In both cases, there is not much gain in performance after a depth ~10.

$\endgroup$
1
  • $\begingroup$ i think you make a fair point that the validation curve is not improving after a certain point, despite increasing complexity of the model. However, i do find it odd that the curve does decrease as the model increasingly overits the training data. It just plateaus $\endgroup$ Feb 9 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.