2
$\begingroup$

I have built an XGBoost classification model in Python on an imbalanced dataset (~1 million positive values and ~12 million negative values), where the features are binary user interaction with web page elements (e.g. did the user scroll to reviews or not) and the target is a binary retail action. My ultimate goal was not so much to achieve a model with an optimal decision rule performance as to understand which user actions/features are important in determining the positive retail action.

Now, I have read quite a bit in forums and literature about evaluating/optimizing an XGBoost model and subsequent decision rule, which I assume is required before achieving my ultimate goal. It seems that there are a lot of different ways to evaluate the decision rule part (e.g. Area Under the Precision Recall Curve, AUROC, etc) and the model (e.g. log-loss). I believe that both AUC and log-loss evaluation methods are insensitive to class balance, so I don't believe that is a concern. However, I am not quite sure which evaluation method is most appropriate in achieving my ultimate goal, and I would appreciate some guidance from someone with more experience in these matters.

Edit: I did also try permutation importance on my XGBoost model as suggested in an answer. I saw pretty similar results to XGBoost's native feature importance. Should I now trust the permutation importance, or should I try to optimize the model by some evaluation criteria and then use XGBoost's native feature importance or permutation importance? In other words, do I need to have a reasonable model by some evaluation criteria before trusting feature importance or permutation importance?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

So your goal is only feature importance from xgboost?

Then don't focus on evaluation metrics, but rather splitting.

I would suggest to read this. Using the default from tree based methods can be slippery.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Noah, Thank you very much for your answer and the link to the information on permutation importance. I can now see I left out some info from my original question. I actually did try permutation importance on my XGBoost model, and I actually received pretty similar information to the feature importances that XGBoost natively gives. This fact did reassure me somewhat. However, I was unsure if permutation importance gives me the best/most accurate answer for my case, or would it be better to optimize the model based on some evaluation metric and then use the native XGBoost feature importance... $\endgroup$
    – Charles
    Dec 30, 2019 at 15:13
  • $\begingroup$ I believe the authors in your linked article are suggesting that permutation importance is the way to go. I will edit my original question for clarification, and I will wait a little longer to see if anyone else has other ideas before marking it as answered. $\endgroup$
    – Charles
    Dec 30, 2019 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.