I am building a binary classification model for imbalanced dataset using XGBoost. I tuned the hyperparameters for four different models based on 2 training datasets and 2 optimization metrics.
- Class Imbalanced dataset; Optimize for Accuracy
- Class Imbalanced dataset; Optimize for F1
- Class balanced dataset; Optimize for Accuracy
- Class balanced dataset; Optimize for F1 Each dataset has 500k Samples
Each of the hyperparameter searches was performed based on 10-folds CV schema. After fitting the models with the best found hyperparameters, each model was evaluated against the same test dataset. I made sure that each training dataset is indeed independent from the test dataset. I also ensured to remove all data just prior to the event of interest, and that the model contains only features that are available to the model at the time of prediction.
I used the following correlation methods to verify that none of the independent features are correlated with the target (dependent variable)
- Pearson correlation (For numerical features)
- spearman correlation (For numerical features)
- Cramér's V correlation (For categorical features)
- theils u (For categorical features)
- correlation ratio (For numerical and categorical features)
While some correlation (>0.9) was found between a few of the independent features, no such correlation was found between any of the independent features and the dependent feature. For the 1st round of model building none of the independent features were removed.
Despite all of this, I was very surprised to find that the model performance on the training and test datasets (for all 4 models) is ‘too good to be true’ on the tuned model.
******************** Trn Results ********************
*** Classification_report:
precision recall f1-score support
0.0 0.959 0.999 0.978 250000
1.0 0.999 0.957 0.977 250000
accuracy 0.978 500000
macro avg 0.979 0.978 0.978 500000
weighted avg 0.979 0.978 0.978 500000
*** accuracy_score: 0.978
*** balanced_accuracy_score: 0.978
*** roc_auc_score: 0.978
*** confusion matrix
TP FP 239242 327
FN TN 10758 249673
*******************************************************
******************** Test Results ********************
*** Classification_report:
precision recall f1-score support
0.0 0.992 0.985 0.989 2275993
1.0 0.860 0.917 0.888 224007
accuracy 0.979 2500000
macro avg 0.926 0.951 0.938 2500000
weighted avg 0.980 0.979 0.980 2500000
*** accuracy_score: 0.979
*** balanced_accuracy_score: 0.951
*** roc_auc_score: 0.951
*** confusion matrix
TP FP 205459 33320
FN TN 18548 2242673
*******************************************************
To check if there is no simple independent features that are directly correlated with the depended feature, I fitted a model with downgraded quality (small max_depth and n_estimators parameters). In this case the model performance for class 1 (rare class) indeed degraded both for the train and test dataset. Which made me believe that the ‘complex’ model is indeed capturing the correct dynamic.
******************** Trn Results ********************
*** Classification_report:
precision recall f1-score support
0.0 0.819 0.844 0.831 250000
1.0 0.839 0.813 0.826 250000
accuracy 0.829 500000
macro avg 0.829 0.829 0.829 500000
weighted avg 0.829 0.829 0.829 500000
*** accuracy_score: 0.829
*** balanced_accuracy_score: 0.829
*** roc_auc_score: 0.829
*** confusion matrix
TP FP 203354 39035
FN TN 46646 210965
*******************************************************
******************** Test Results ********************
*** Classification_report:
precision recall f1-score support
0.0 0.979 0.845 0.907 2275993
1.0 0.340 0.812 0.480 224007
accuracy 0.842 2500000
macro avg 0.659 0.829 0.693 2500000
weighted avg 0.921 0.842 0.869 2500000
*** accuracy_score: 0.842
*** balanced_accuracy_score: 0.829
*** roc_auc_score: 0.829
*** confusion matrix
TP FP 181928 352543
FN TN 42079 1923450
*******************************************************
Am I correct to assume that there is some type of data leakage here? How can I identify it?
