# XGBOOST missing_value feature degrades my performance?

I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction means is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

1. the cross-validated performance on the train population stays the same.
2. BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?

• What is the percentage of missing values? Aug 6 '17 at 14:59
• @Pieter - about 60% missing values in the train set and 57% missing values in the test set. It varies of course between features according to their popularity - goes from 14% missing for BloodPressure to almost 99% missing for PlasmaAnionGap Aug 7 '17 at 6:28
• This does seem very strange. Especially the prediction mean change. I know this is years after OP, but some additional information could have helped: Is -65336 less than all of your real data values? You're setting missing_value and retraining, right? Which flavor (CLI/R/Python/etc) are you using? (And what version; this may have since been rectified.) Dec 3 '19 at 2:59