help interpreting training/validation curves for classification tree

Question

I'm developing a binary classification tree and having some touble interpreting my training/validation curves. I used the CART algorithm with information gain as my splitting criterion. The training and test data was split 75% : 25%. There was an imbalance between my two classes, so i performed stratified (proportionate) sampling during the train/test split. Afterwards, I oversampled the minority class in my training set.

Below, i'm increasing the depth of the tree and observing the performance in terms of F1 score (harmonic mean of precision and recall; higher is better).

as the complexity of tree increases, my training curve approaches zero error (as expected) while my validation curve plateaus. I would expect the validation curve to decrease as the tree overfits, but here the validation curve seems to peak just as the tree is maximally overfitting the training data. Is this an indication of something I'm doing wrong, or is there a reasonable interpretation for this?

UPDATE:

I refit the tree, this time without oversampling minority class. I now have training/validation curves that exhibit more typical behavior.

the validation curve peaks before the tree overfits, and max performance is the same as before (~0.84), only this time with a tree that has half the complexity (depth of 14 vs 25). I am starting to think oversampling is not actually helping the tree learn.

Answer 1

You are correct that you are overfitting (i.e., a large drop in performance between training and hold-out dataset performance).

Given that decision tree algorithms are high variance, you are looking too closely at hold-out dataset performance. In both cases, there is not much gain in performance after a depth ~10.