Let's say we have a categorical feature $X_i$ and we have build a black-box classification model like xgboost with $X_i$ as one of many predictors. We'd like to ask a question: does $X_i$ affects the overall prediction and, if so, how much?
In particular $X_i$ could be:
- a dichotomous variable
- a n-level variable where we are interested in the potential difference between two particular levels
In white-box models like linear regression we have tests to obtain statistical significance. But can we obtain statistical-significance-alike with black box models? Does any tool from explainable artifficial intelligence field is applicable to that? Or would it be better to just perform standard t-test on the output probabilities predictions?