1
$\begingroup$

I'm training an xgboost binary classification model. The data I have is around 600k and positive is only 0.1% of it. I tried to use all overfitting prevention techniques xgboost has to offer (tune eta, gamma, min_child_weight, subsample feature/data, early stopping etc). However, my model either strongly overfits but with decent metric on test set or does not overfit too strong but general performances are bad. I've listed two training results (metric is auc-pr) here,

AUC-PR 130 trees 3800 trees
train 0.18 0.98
test 0.05 0.45
validation(early stopping) 0.06 0.46

I checked for data leakage but doesn't seem to have any. My question is:

  • Can I use the model with 3800 trees, even though it's overfitting, it's getting decent results on test set?
  • Do I absolutely have to be strict in preventing overfitting for this strong class imbalance scenario? If so, what are other techniques?
  • Anything I did wrong?
$\endgroup$
4
  • 1
    $\begingroup$ Welcome to DataScienceSE. The two models are overfit but the 3800 trees one not only has better perf, it is actually less overfit based on the ratio test perf / train perf. Did you try resampling? $\endgroup$
    – Erwan
    Jan 13 at 0:38
  • $\begingroup$ @Erwan Thank you! I'm leveraging scale_pos_weight in xgboost. $\endgroup$
    – HanaKaze
    Jan 13 at 0:41
  • 1
    $\begingroup$ Is there any heuristic on what you mention? ratio test perf / train perf $\endgroup$
    – Moreno
    Jan 13 at 1:02
  • 1
    $\begingroup$ @Moreno: no not at all, it's just an interpretation on my part. What I meant is mostly that there's no reason to see the second model as more overfit based on the larger difference, they are both overfit anyway. $\endgroup$
    – Erwan
    Jan 13 at 9:06
1
$\begingroup$

With complex data it's rare not to have any overfitting (or underfitting). Ideally one wants to avoid strong overfitting, and if given a choice between two models it's clearly safer to use one which isn't overfit. But if it's impossible to avoid, from a practical perspective there's no reason not to use a model for this reason, in my opinion. The problem with an overfit model is simply that it's going to perform poorly on fresh instances, so the worst that can happen is the performance that you observe on the test set.

Just an idea: I don't know if it makes sense with your data but it might be worth trying one-class classification on the minority class. This could be a way to avoid the imbalance issue, and could be followed by a second stage of regular binary classification.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.