How to interpret the AUC score in this case?

I just run a random forest model on a imbalance dataset. I got the set of AUC and the confusion matrix. The AUC seemed not bad but actually the model predict every instance as positive. So how it happened and how to use AUC properly?

The ROC Curve as below:

I plot out the predicted probability of positive class in test set. The probability was within a tight range (0-0.4).

2 Answers

If you want to figure it out how this ROC happens , you would better LIST the tuples including your "predicted" values and the "truth" values, and SORT with the "predicted" value , then PLOT the ROC .

In your case , the tuples and points should be like this :

  predicted truth (x,y)
0.53 0  (6/6,14/14)
0.55 0  (5/6,14/14)
0.57 1  (4/6,14/14)
0.59 0  (4/6,13/14)
0.60 1  (3/6,13/14)
0.62 1  (3/6,12/14)
0.63 0  (3/6,11/14)
0.64 0  (2/6,11/14)
0.66 1  (1/6,11/14)
0.68 1  (1/6,10/14)
0.71 1  (1/6,9/14)
0.73 1  (1/6,8/14)
0.77 1  (1/6,7/14)
0.78 0  (1/6,6/14)
0.82 1  (0/6,6/14)
0.86 1  (0/6,5/14)
0.89 1  (0/6,4/14)
0.92 1  (0/6,3/14)
0.94 1  (0/6,2/14)
0.96 1  (0/6,1/14)
(0/6,0/14)


Put these points (x,y) on ROC picture , it should be like this :

Just much like that of yours !

By the way , if you want to know how these points are figured out , you can check the code written in scala in the below :

    def computeAuc(predict: BDV[Double],groundTruth: BDV[Double]): Double
= {

// retrieve number of positive and negative samples in ground truth
val nPos = groundTruth.toArray.filter(_>0).length
val nNeg = groundTruth.toArray.filter(_<=0).length

// tuple predict with ground truth , and sort with predict
val pair = predict.toArray.zip(groundTruth.toArray)
val sortedPair = pair.sortBy(_._1)
var auc = 0.0.toDouble
val x = BDV.zeros[Double](predict.length + 1)
val y = BDV.zeros[Double](predict.length + 1)
x(0) = 1.0
y(0) = 1.0

// calculate auc incrementally
var i = 1.toInt
while(i < sortedPair.length) {
y(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 > 0).length) / nPos
x(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 <= 0).length) / nNeg
auc = auc + (((y(i) + y(i - 1))*(x(i - 1) - x(i)))/2.0)
i += 1
}
auc = auc + ((y(i - 1) * x(i - 1))/2.0)

auc
}


Finally , your imbalance problem is severe , and you would better do down-sampling or up-sampling before training .

Hopes this contributes you , good luck -)

• Thank you for response and quite nice presentation. Inspiring. Commented Dec 16, 2016 at 0:38

AUC is based on rank order of your predictions, not the actual class to which it's assigned. It's very likely that the scale of the output is misbehaving.

Look at the values of your predictions, I suspect that the predictions of your model are within a tight range. If that's the case, the argmax will yield the same class for all of your observations (which is what's happening).

You may wish to tinker with some of the hyperparameters to see which one is causing this exactly (might start with the learning rate). It's probably worth testing if a logistic regression gives you the same problem, which will help identify whether or not it's a problem with your inputs/features.

• Thanks. Please see above. I used the random threshold 0.5. So the model predicted zero for all test points. Commented Dec 14, 2016 at 3:08
• Actually, it doesn't look so bad but all of the predictions are below 0.5, which is what's driving the class to be assigned to 0. I'd probably recommend down sampling the 0 classes or weighting your data by the distribution of your classes. Commented Dec 14, 2016 at 3:10
• How about oversampling or change the threshold? Commented Dec 14, 2016 at 3:22
• Changing the threshold can work, too, but it's not as satisfying of a solution. Oversampling is what I was saying in my last comment. That should help quite a bit but give the logistic a try too. Commented Dec 14, 2016 at 3:28