1
$\begingroup$

I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)

I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.

Is there a way to debug and understand which of the model assumptions which held at A do not hold at B? I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.

Would that help me understand what broke? or would I just get tons of tiny differences? Is there some other simple way I'm missing?

(related to this survey Machine learning learn to work well on future data distribution?)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Not sure if you've tried this already, but you might dig into XGBOOST feature importance to establish how your model is making predictions, then do a more in-depth comparison between the two populations at those splits.

This isn't very different from what you've proposed, but it does make for a more focused analysis; you might have time to look at higher order interactions between the more important features. (I'm assuming it's not possible to cheat by training a classifier on your population B and comparing how each model makes predictions!)

ELI5 and LIME also come to mind for debugging feature importance and explaining model prediction.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks, but the classifier I'm talking about is already using only the 30 most important features, I selected them by building a bigger model, and then building a more focused classifier using the 30 top feature_importances rank $\endgroup$
    – ihadanny
    Commented Mar 11, 2018 at 9:17
0
$\begingroup$

I would train new models and use that to partition the samples, then do a exploratory data analysis on these sets of samples.

For instance train model B-only on population B and look at samples intersection(Aonly_wrong, Bonly_correct). May also want train a model on A+B, then compare.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.