I'm training an
xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of
0.90. I'm using a special value
-65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.
I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC
0.84, and my prediction means is a reasonable
THEN, I tell XGBOOST that
- the cross-validated performance on the train population stays the same.
- BUT the performance on the general population degrades from AUC
0.79. Also, I noticed that the prediction mean is changing from
Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?