1
$\begingroup$

I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000. All positive examples are correctly predicted, instead negatives elements correctly predicted are 99% of total negatives.

Plot of PR and ROC curves are those:

What can be inferred from these curves? Those are my firsts works with classifiers and i'm confused. I think that precision is always low, because the negatives that are wrong predicted as positive have an high score assigned by the classifier (close to 1). ROC instead i think that is high because all positive examples are correctly predicted. These are my suppositions, correct me if I am wrong.

$\endgroup$
1
$\begingroup$

For binary-class classification problems and data being highly unbalanced,

  1. Go for AUC and f1 score as metrics.
  2. Plot the confusion matrix.
  3. Split the data into train:valid:test::60:20:20 or 80:10:10, and do Cross-validation and hyperparameter tuning on train and valid sets. Then go for test set.
  4. You could also try bootstrap resampling.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.