I have an unbalanced data set with about 8% of negative examples. The goal is to minimize false negatives given a cost matrix. It seems like SVM (with radial kernel) and random forest work best. How should I tune hyperparameters in this setting?

My suggestion: separate data into train/validation set, use probabilistic output together with cost matrix to assign predicted classes, tune hyperparameters to maximize accuracy.

How can I increase performance? Currently I use random forest with nodesize=1 and mtry=5 which gives about 97% accuracy.

  • $\begingroup$ How do you define/calculate accuracy? $\endgroup$ – Diego Mar 17 '16 at 11:24

Tuning the hyperparameters of a model is still more of an art than a science.

Both R (caRet) and Python (sklearn) have frameworks for performing grid searches of hyperparameters using cross validation. You can define custom cost functions in these frameworks, which will allow you to directly assess your cross-validation results using your cost matrix. See the scikit-learn documentation for Python, or the caRet documentation for R. You will get much better results using your cost matrix directly to evaluate your models, instead of using F-1, precision, recall, accuracy, or any other metric.

Other Guidance

It is important to understand what the different hyperparameters actually mean in the context of each learning algorithm to help guide the parameter search, as computational limitations usually strictly limit the number of different models you can build total, limiting the size of the grid you can realistically search.

With such a low percentage of negative examples in your set, you will want to ensure that you use stratified sampling for your cross-validation. This will ensure that each fold has the same percentage of negative examples as the base set. Otherwise, your cross-validation results will likely be biased by the non-uniform negative class distribution.


I solve this task as follows: cross-validate train data in order to maximize F1-score then use probabilistic output and cost matrix for predicting a class for a new item. Any suggestions are welcome.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.