1
$\begingroup$

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead negatives elements correctly predicted are 99% of total negatives.

Plot of PR and ROC curves are those:

What can be inferred from these curves? Those are my firsts works with classifiers and i'm confused. I think that precision is always low, because the negatives that are wrong predicted as positive have an high score assigned by the classifier (close to 1). ROC instead i think that is high because all positive examples are correctly predicted. These are my suppositions, correct me if I am wrong.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

For binary-class classification problems and data being highly unbalanced,

  1. Go for AUC and f1 score as metrics.
  2. Plot the confusion matrix.
  3. Split the data into train:valid:test::60:20:20 or 80:10:10, and do Cross-validation and hyperparameter tuning on train and valid sets. Then go for test set.
  4. You could also try bootstrap resampling.
Improve this answer
$\endgroup$
0
$\begingroup$

With such imbalanced data, the area under the ROC curve is not really informative. Area under the precision recall curve is better.

Improve this answer
$\endgroup$
1
  • $\begingroup$ In what way does the ROC curve get misled by the imbalance? $\endgroup$
    – Dave
    Jan 11, 2022 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.