I have recently been trying to train a randomForest model on a binary outcome with a very uneven class split.
282 control ~82% 63 case ~18%
There are a total of 147 predictors that I'm testing for this model.
I have been running Monte Carlo cross validations to determine the best mtry parameter however I have ran into an odd issue.
Based on these plots it looks like my classifier is no better than random (when you consider the class imbalance). The part that confuses me is that for some reason Kappa seems to increase as Accuracy decreases.. which doesn't make a ton of sense to me. Furthermore the graphs seem to suggest that the best mtry in terms of the kappa statistic is equal to the number of predictors, which I am having trouble interpreting the significance of. Does this mean my model is not useful at all for feature importance selection?