I'm training an xgboost binary classification model. The data I have is around 600k and positive is only 0.1% of it. I tried to use all overfitting prevention techniques xgboost has to offer (tune eta, gamma, min_child_weight, subsample feature/data, early stopping etc). However, my model either strongly overfits but with decent metric on test set or does not overfit too strong but general performances are bad. I've listed two training results (metric is auc-pr) here,
|AUC-PR||130 trees||3800 trees|
I checked for data leakage but doesn't seem to have any. My question is:
- Can I use the model with 3800 trees, even though it's overfitting, it's getting decent results on test set?
- Do I absolutely have to be strict in preventing overfitting for this strong class imbalance scenario? If so, what are other techniques?
- Anything I did wrong?