I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative).

I've tried various algorithms (Naive Bayes, Random Forest, AODE, C4.5) and all of them have unacceptable false negative ratios. Random Forest has the highest overall prediction accuracy (99.5%) and the lowest false negative ratio, but still misses 79% of positive classes (i.e. fails to detect 79% of malignant tumors).

Any ideas how I can improve this situation?

Thanks!

Class imbalance is a very common problem. You can either oversample the positive class (or undersample the negative) or add class weights.

Another thing to remember in this case is that accuracy is not a very useful metric here. You might consider AUC or F1 score.

Changing your decision threshold may seem appealing, but will obviously lead to (in this case likely drastically) increased false positives (though perhaps FPs aren't as bad as FNs in the case of medical diagnosis, if tests aren't prohibitively expensive).

A more in-depth analysis of options in the case of class imbalance is provided here.

Even though the answer in reality is always true or false, you can make your class attributes not labels but floating point numbers, ie 1.0 and 0.0 (or 100 and 0). That is, you can frame it is a regression problem, not classification problem.

Then the predicted output will likewise be numbers on that spectrum, ie probabilities not labels. Your current implementation is essentially equivalent to a regression model with a threshold of 0.5.

With such an output, you or your can client can define a threshold that is acceptable (eg 0.3). Of course there will be more false positives then, but for some applications, like detecting cancer, that is optimal.

Oversampling (the positives) or undersampling (the negatives) are also ways to fix this, but must be done thoughtfully, can sacrifice accuracy, and still sacrifices the control to move the threshold after creating the data and training.

You can have a look at this question where I got interesting replies for this problem.

Best regards

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.