Unbalanced classes -- How to minimize false negatives?

Question

I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative).

I've tried various algorithms (Naive Bayes, Random Forest, AODE, C4.5) and all of them have unacceptable false negative ratios. Random Forest has the highest overall prediction accuracy (99.5%) and the lowest false negative ratio, but still misses 79% of positive classes (i.e. fails to detect 79% of malignant tumors).

Any ideas how I can improve this situation?

Thanks!

Answer 1

Class imbalance is a very common problem. You can either oversample the positive class (or undersample the negative) or add class weights.

Another thing to remember in this case is that accuracy is not a very useful metric here. You might consider AUC or F1 score.

Changing your decision threshold may seem appealing, but will obviously lead to (in this case likely drastically) increased false positives (though perhaps FPs aren't as bad as FNs in the case of medical diagnosis, if tests aren't prohibitively expensive).

A more in-depth analysis of options in the case of class imbalance is provided here.

Answer 2

Even though the answer in reality is always true or false, you can make your class attributes not labels but floating point numbers, ie 1.0 and 0.0 (or 100 and 0). That is, you can frame it is a regression problem, not classification problem.

Then the predicted output will likewise be numbers on that spectrum, ie probabilities not labels. Your current implementation is essentially equivalent to a regression model with a threshold of 0.5.

With such an output, you or your client can define a threshold that is acceptable (eg 0.3). Of course there will be more false positives then, but for some applications, like detecting cancer, that is optimal.

Oversampling (the positives) or undersampling (the negatives) are also ways to fix this, but must be done thoughtfully, can sacrifice accuracy, and still sacrifices the control to move the threshold after creating the data and training.

Answer 3

Under- and over-sampling as a technique have already been mentioned, but I thought I would point to a commonly used variant:

SMOTE: Synthetic Minority Over-sampling Technique

It was presented in this paper in 2002. Here is a snippet from the abstract:

This paper shows that a combination of our method of over-sampling the minority (abnormal) class and under-sampling the majority (normal) class can achieve better classifier performance (in ROC space) than only under-sampling the majority class.

---

You can use it easily in Python, using the imbalanced-learn package, which is contained in the contrib module of Scikit-Learn and must be installed separately.

imbalanced-learn is a python package offering a number of re-sampling techniques commonly used in datasets showing strong between-class imbalance.

That package includes methods for combining over-/under-sampling as well as a set of utilities to generate batches of data that can flow into Keras/Tensorflow.