I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)
I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.
Is there a way to debug and understand which of the model assumptions which held at A do not hold at B? I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.
Would that help me understand what broke? or would I just get tons of tiny differences? Is there some other simple way I'm missing?
(related to this survey Machine learning learn to work well on future data distribution?)