I have a supervised binary classification problem. I tuned an xgboost model on the training set and achieved a reasonably high accuracy on the test set. Now I want to interpret the results of the model.

I used the SHAP library to interpret the results and for the most part, they are consistent with what I would expect. However, there is one feature that, if we average over all shap values, is ranked as 7th most important and I would have expected it to have been higher. If I perform a t-test of the feature between the positive group and the negative group, there is a clear statistical difference between the two (p<<0.05) which implies that the feature should be very predictive of the class. What could be the cause of this discrepancy?

  • $\begingroup$ t test uses a lot of assumptions and can lead to unreliable results. specifically it sounds like the positive and negative group are not independent since they are from the same training set $\endgroup$ May 17, 2022 at 15:55
  • $\begingroup$ I believe they are independent since the data extracted comes from different mice for the positive and negative groups. Perhaps it is a correlation issue? Highly correlated variables in the input matrix may mask the importance of one in a shap analysis since one of them may not be used by the model (due to redundant information)? $\endgroup$
    – Evan
    May 17, 2022 at 16:35
  • $\begingroup$ Possibly. I'm not familiar with the SHAP package, but might make sense to run a logistic regression and see if the coefficients roughly equate to what you are seeing. Or random forest feature importance $\endgroup$ May 17, 2022 at 23:32


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.