I'm training a XGBOOST model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction mean is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

1. the cross-validated performance on the train population stays the same.
2. BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?

I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction means is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

1. the cross-validated performance on the train population stays the same.
2. BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?