I have a side project where I am doing credit scoring using R (sample size around 16k for train data and 4k for test data, and also another two 20k data batches for out-of-time validation) with unbalanced classes (3.5% of BAD type of clients).
I'm trying to make various models to have enough of them to make ensembles, but for this purpose, let's focus on one of them, particularly XGBoost.
I was reading a lot on how to tackle overfitting, but I haven't found any good source on how to do it in practice, step-by-step. As for now, my two best models have:
0.8 AUC on training data, around 0.799 AUC on holdout set and around 0.7355 and 0.7195 AUC on out-of-time batches.
0.764 AUC on training, 0.7785 AUC on the holdout test set and 0,7285 AUC on both out-of-time batches.
I am worried about is that drop on out-of-time batches, since I think that 0.05-0.08 drop is huge and it might be a sign that models that I did, really are overfitting and don't generalize well. To clarify, while I was tweaking models, I didn't know about those out-of-time scores.
Could anyone share the experience what is best practice to detect overfitting? And does those two models overfit, or I am just panicking, and this drop in performance is normal?
My current pipeline in general looks like this:
Feature importance using xgboost package to take best 300 features from all 2400 available.
Removing highly-correlated features (0.75 cutoff) - 123 features left
Train/test split - 0.8 vs 0.2 plus two out-of-time batches
Model selection using nested CV(5-fold CV in outer) with hyperparameter tuning in inner loop(5-fold CV in inner) - all done in MLR package.
From 5 models I get from nested CV, I'm picking the best performing one (that has the closest AUC in both train and holdout test set)
And then when I was happy with the model I performed a test on out-of-time models.
How I could improve my pipeline, in a way that I could detect overfitting? Is there any list of steps that would roughly cover what it could be done to reduce it?
Also, in highly unbalanced case, choosing a good validation set means that I only need to take care of the proper distribution of the target variable? Should I take care of something else as well?