4
$\begingroup$

I've created a model with Random Forest algorithm. There are 45k observations, where 1s I have 12% and the rest are 0s. As far as I know ROC AUC is not the best evaluation metric in such a case. I went with PR AUC and got 59%. How would you assess the results ?

classification report and confusion matrix

PR AUC

ROC AUC

New contributor
Tomasz Przybyło is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

2 Answers 2

2
$\begingroup$

As discussed on the stats.SE meta, class imbalance leads to a lot of misconceptions, and it is important to have a strong understanding of the underlying statistics in order to overcome those misconceptions.

Log loss and Brier score are two popular metrics, whether there is imbalance or not. Among their advantages is that they are strictly proper scoring rules that are uniquely optimized in expected value by the true probabilities. (Links contained in that stats.SE meta post discuss the importance of such scoring rules.)

Let $y_i$ and $p_i$ be the $i^{th}$ true observation ($0$ or $1$) and predicted probability, respectively.

$$ \text{Log Loss}\\ -\dfrac{1}{N}\sum_{i=1}^N\bigg[ y_i\log(p_i)+(1-y_i)\log(1-p_i) \bigg] $$

$$ \text{Brier Score}\\ \dfrac{1}{N} \sum_{i=1}^N\bigg( y_i-p_i \bigg)^2 $$

These are related to the McFadden and Efron pseudo $R^2$ values discussed at a nice UCLA page.

$\endgroup$
2
  • $\begingroup$ Any explanation for the downvote? $\endgroup$
    – Dave
    Sep 21 at 15:30
  • 2
    $\begingroup$ Can't explain the downvote but gave you an upvote ;) $\endgroup$
    – Erwan
    Sep 21 at 15:34
0
$\begingroup$

First, as far as I know a ROC curve and AUC can perfectly be used with imbalanced (binary classification) data.

However I think there is a problem with the ROC curve you show at the end: it seems to be calculated with the majority class (0) as positive class? Maybe there's a bug in your code.

The PR plot doesn't seem to have this problem, it correctly shows the Precision/Recall for class 1.

The results are fine, they don't show any clear issue. But knowing if this a good performance or not depends on the task/data.

$\endgroup$
2
  • $\begingroup$ Regarding ROC I think it's due to imbalanced data $\endgroup$ Sep 21 at 19:59
  • $\begingroup$ @TomaszPrzybyło no, this is not due to imbalanced data. As I said, the ROC curve shows very good performance but it's incorrect because it should show TPR and FPR for class 1 (small class, harder), not class 0 (large class, easier). The ROC curve should be much closer to the diagonal. $\endgroup$
    – Erwan
    Sep 21 at 22:29

Your Answer

Tomasz Przybyło is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.