I am building a predictive model designed to predict attrition within my organization. I am trying to decide whether to add certain predictors to my model. I used a Kruskal-Wallis rank sum test to check the correlation between a few of my predictor variables and my response variable and found that the predictors are independent from my response. Should I still include these variables in my model? I am leaning towards no due to the lack of correlation with my response variable but don't want to discard a potential "split" in my decision tree. My data set contains 8,225 observations and 173 columns which I have been using as predictors.
In general, a Kruskal-Wallis test (or any other univariate test) cannot guarantee that there are no useful interactions with other features that a decision tree might still pick up. That means you won't know for sure until you try it.
You can test the performance with and without these "weak" features and compare results. Make sure you control for overfitting when including all features. Alternatively, after fitting the decision tree, you can look at the target response when varying the feature values ("partial dependence").
When developing a prediction model, I typically prefer to let the model decide what is most important for prediction. If you have enough data, the model should learn to disregard unimportant predictors; but watch carefully for overfitting, as recommended by @oW_.
A few thoughts:
Statistical tests often make assumptions. One of the benefits of tree methods is that they can model relationships without being constrained by assumptions.
A predictor may appear unimportant on its own but it may be important in combination with other predictors, as alluded to by @oW_.
Another benefit of tree methods is that they can make use of many weak predictors to produce a strong predictor.
In the case of prediction, unless you have strong reasons, I find that one loses more than one gains by trying to hand-select predictors.