1
$\begingroup$

I just run a random forest model on a imbalance dataset. I got the set of AUC and the confusion matrix. The AUC seemed not bad but actually the model predict every instance as positive. So how it happened and how to use AUC properly?

enter image description here

The ROC Curve as below:

enter image description here

I plot out the predicted probability of positive class in test set. The probability was within a tight range (0-0.4).

enter image description here

$\endgroup$
2
$\begingroup$

If you want to figure it out how this ROC happens , you would better LIST the tuples including your "predicted" values and the "truth" values, and SORT with the "predicted" value , then PLOT the ROC .

In your case , the tuples and points should be like this :

  predicted truth (x,y)
    0.53 0  (6/6,14/14)
    0.55 0  (5/6,14/14)
    0.57 1  (4/6,14/14)
    0.59 0  (4/6,13/14)
    0.60 1  (3/6,13/14)
    0.62 1  (3/6,12/14)
    0.63 0  (3/6,11/14)
    0.64 0  (2/6,11/14)
    0.66 1  (1/6,11/14)
    0.68 1  (1/6,10/14)
    0.71 1  (1/6,9/14)
    0.73 1  (1/6,8/14)
    0.77 1  (1/6,7/14)
    0.78 0  (1/6,6/14)
    0.82 1  (0/6,6/14)
    0.86 1  (0/6,5/14)
    0.89 1  (0/6,4/14)
    0.92 1  (0/6,3/14)
    0.94 1  (0/6,2/14)
    0.96 1  (0/6,1/14)
            (0/6,0/14)

Put these points (x,y) on ROC picture , it should be like this : enter image description here

Just much like that of yours !

By the way , if you want to know how these points are figured out , you can check the code written in scala in the below :

    def computeAuc(predict: BDV[Double],groundTruth: BDV[Double]): Double
    = {

        // retrieve number of positive and negative samples in ground truth
        val nPos = groundTruth.toArray.filter(_>0).length
        val nNeg = groundTruth.toArray.filter(_<=0).length

        // tuple predict with ground truth , and sort with predict
        val pair = predict.toArray.zip(groundTruth.toArray)
        val sortedPair = pair.sortBy(_._1)
        var auc = 0.0.toDouble
        val x = BDV.zeros[Double](predict.length + 1)
        val y = BDV.zeros[Double](predict.length + 1)
        x(0) = 1.0
        y(0) = 1.0

        // calculate auc incrementally
        var i = 1.toInt
        while(i < sortedPair.length) {
          y(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 > 0).length) / nPos
          x(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 <= 0).length) / nNeg
          auc = auc + (((y(i) + y(i - 1))*(x(i - 1) - x(i)))/2.0)
          i += 1
        }
        auc = auc + ((y(i - 1) * x(i - 1))/2.0)

        auc
   }

Finally , your imbalance problem is severe , and you would better do down-sampling or up-sampling before training .

Hopes this contributes you , good luck -)

$\endgroup$
  • $\begingroup$ Thank you for response and quite nice presentation. Inspiring. $\endgroup$ – LUSAQX Dec 16 '16 at 0:38
0
$\begingroup$

AUC is based on rank order of your predictions, not the actual class to which it's assigned. It's very likely that the scale of the output is misbehaving.

Look at the values of your predictions, I suspect that the predictions of your model are within a tight range. If that's the case, the argmax will yield the same class for all of your observations (which is what's happening).

You may wish to tinker with some of the hyperparameters to see which one is causing this exactly (might start with the learning rate). It's probably worth testing if a logistic regression gives you the same problem, which will help identify whether or not it's a problem with your inputs/features.

$\endgroup$
  • $\begingroup$ Thanks. Please see above. I used the random threshold 0.5. So the model predicted zero for all test points. $\endgroup$ – LUSAQX Dec 14 '16 at 3:08
  • $\begingroup$ Actually, it doesn't look so bad but all of the predictions are below 0.5, which is what's driving the class to be assigned to 0. I'd probably recommend down sampling the 0 classes or weighting your data by the distribution of your classes. $\endgroup$ – franciscojavierarceo Dec 14 '16 at 3:10
  • $\begingroup$ How about oversampling or change the threshold? $\endgroup$ – LUSAQX Dec 14 '16 at 3:22
  • $\begingroup$ Changing the threshold can work, too, but it's not as satisfying of a solution. Oversampling is what I was saying in my last comment. That should help quite a bit but give the logistic a try too. $\endgroup$ – franciscojavierarceo Dec 14 '16 at 3:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.