Strong overfitting accompanying strong class imbalance

Question

I'm training an xgboost binary classification model. The data I have is around 600k and positive is only 0.1% of it. I tried to use all overfitting prevention techniques xgboost has to offer (tune eta, gamma, min_child_weight, subsample feature/data, early stopping etc). However, my model either strongly overfits but with decent metric on test set or does not overfit too strong but general performances are bad. I've listed two training results (metric is auc-pr) here,

AUC-PR	130 trees	3800 trees
train	0.18	0.98
test	0.05	0.45
validation(early stopping)	0.06	0.46

I checked for data leakage but doesn't seem to have any. My question is:

• Can I use the model with 3800 trees, even though it's overfitting, it's getting decent results on test set?
• Do I absolutely have to be strict in preventing overfitting for this strong class imbalance scenario? If so, what are other techniques?
• Anything I did wrong?

Answer 1

With complex data it's rare not to have any overfitting (or underfitting). Ideally one wants to avoid strong overfitting, and if given a choice between two models it's clearly safer to use one which isn't overfit. But if it's impossible to avoid, from a practical perspective there's no reason not to use a model for this reason, in my opinion. The problem with an overfit model is simply that it's going to perform poorly on fresh instances, so the worst that can happen is the performance that you observe on the test set.

Just an idea: I don't know if it makes sense with your data but it might be worth trying one-class classification on the minority class. This could be a way to avoid the imbalance issue, and could be followed by a second stage of regular binary classification.