a simple way to test wether a tree-based classifier would transfer well to a target population?

I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)

I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.

Is there a way to debug and understand which of the model assumptions which held at A do not hold at B? I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.

Would that help me understand what broke? or would I just get tons of tiny differences? Is there some other simple way I'm missing?

(related to this survey Machine learning learn to work well on future data distribution?)

• thanks, but the classifier I'm talking about is already using only the 30 most important features, I selected them by building a bigger model, and then building a more focused classifier using the 30 top feature_importances rank – ihadanny Mar 11 '18 at 9:17
For instance train model B-only on population B and look at samples intersection(Aonly_wrong, Bonly_correct). May also want train a model on A+B, then compare.