Using random forest to learn Imbalanced Data (rare disease)

Question

I met a question when I ran the random forest. I used "V1", "V2", "V3" to predict a binary outcome (1: sick; 0: no) with random forest.

I got a very high accuracy score (99%) however, when I check the confusion matrix, it shows that none of sick individuals were caught in testing data set (30% of entire data set). Here is the confusion matrix:

[[856 0]

[ 9 0]]

This result means that 0 out of 9 people was detected as sick and it causes my attention. Maybe because the data set is imbalanced (very few sick individuals)?

I would like to see if there is any other ways to detect sick individuals rather than a high accuracy rate, which means it is OK it has higher false positive rate but I would like to catch all 9 (true positive) individuals.

Thanks!

Answer 1

Use class weights, to weight errors, so that "incorrectly labelling a sick person as healthy" is penalized more than "incorrectly labelling a healthy person as sick". Or look up any of the other standard techniques for dealing with class imbalance.

Answer 2

I would pick a different scoring function than accuracy; the problem with accuracy is that if you classify all the instances under the majority class, you will automatically end up with a very high accuracy score, which is rather meaningless!

Usually, the area under the curve (AUC) of the precision-recall curve (sklearn.metrics.average_precision_score() in Python) works well and is representative of the actual performance of the model when one is dealing with unbalanced data. The AUC of the receiver operating characteristic (ROC) curve is also another metric that is most often used.

Having said this, it seems to me that you specifically want to maximize the recall score, which is the ratio of the number of true positives to the total number of actual positives.

EDIT: As per @stmax's comment below, you don't want to maximize the recall score either.