1
$\begingroup$

I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction means is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

  1. the cross-validated performance on the train population stays the same.
  2. BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ What is the percentage of missing values? $\endgroup$
    – Pieter
    Commented Aug 6, 2017 at 14:59
  • $\begingroup$ @Pieter - about 60% missing values in the train set and 57% missing values in the test set. It varies of course between features according to their popularity - goes from 14% missing for BloodPressure to almost 99% missing for PlasmaAnionGap $\endgroup$
    – ihadanny
    Commented Aug 7, 2017 at 6:28
  • $\begingroup$ This does seem very strange. Especially the prediction mean change. I know this is years after OP, but some additional information could have helped: Is -65336 less than all of your real data values? You're setting missing_value and retraining, right? Which flavor (CLI/R/Python/etc) are you using? (And what version; this may have since been rectified.) $\endgroup$
    – Ben Reiniger ♦
    Commented Dec 3, 2019 at 2:59

1 Answer 1

Reset to default
-1
$\begingroup$

If you have more than 50% of values as missing across any columns you should impute only when you are an SME of the domain or know the reason for missing(Honestly I would drop that column rather imputing). Not only XGboost most of the algorithm would penalise for Missing values and one can increase the performance by imputation. Tip may be try Catboost

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.