I am currently using Xgboost for a binary classification problem with highly imbalanced data in R.
I have used oversampling to train the model. This worked well, now however it comes to measuring variable importance - how does the expected loss change when a certain variable is randomized (this is one of the methods the DALEX package provides).
This has lead to the following: When I compare the variable importance between two models- one trained on a oversampled set and one trained on the original set - I get vastly different results.
It is not surprising that the actual values change, but why do some variables gain and others lose importance as measured above?