1
$\begingroup$

I was training model on a very imbalanced dataset with 80:20 ratio of two classes. The dataset has thousands of rows and I trained the model using

DeccisionTreeClassifier(class_weight='balanced')

The precision and recall I get on the test set were very strange

Test set precision : 0.987767
Test set recall : 0.01432

I'm unable to interpret the results. What does this tell me about my classifier?

Improve this question
$\endgroup$
3
  • $\begingroup$ Many of my datasets are in the >99 to <1 ratio. Is the cutoff used for precision and recall in scikit for your code the optimal cutoff for your business problem? Before looking at the confusion matrix stats, you should know your optimal cutoff and make the confusion matrix from that level. Even so, when we get very imbalanced, the confusion matrix may not be the best way to examine performance. $\endgroup$
    – Craig
    Commented Jun 1, 2020 at 10:06
  • $\begingroup$ @Craig I'm not sure what you mean by cutoff and what metric would be the preferable metric to examine performance in such case? $\endgroup$
    – skrrrt
    Commented Jun 1, 2020 at 10:15
  • 1
    $\begingroup$ Some classifiers output a well calibrated probability, some a distance, some a logit. Let's assume the score is a probability. The cutoff is the probability value that score >= is a predicted 1 (event) and < is a predicted 0 (non-event). What value is right for your problem? Think of it like business_value(TP+TN) - business_costs(FP+FN). There is a cost to being wrong and a value to being correct. Predicting if a program is a virus vs winner of a game. A 0.50 may not be the optimal cutoff for the business problem and the model. Examining a confusion matrix at the wrong cutoff is not optimal. $\endgroup$
    – Craig
    Commented Jun 1, 2020 at 11:35

1 Answer 1

Reset to default
1
$\begingroup$

All problems due to "a very imbalanced dataset". Please look at the definition of recall and precision. Based on your score I could say that you a very small set of values labeled as positive, which are classified correctly($precision=\frac{TP}{TP+FP}$). But you have a very big set of values labeled as negative, which have influence on $recall=\frac{TP}{TP+FN}$, in that way that $TP$ stays the same like in precision, but have you a lot of $FN$ values which leads to small value of recall.

At least you should resample of the dataset to make better results.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I was under the impression that class_weight = balanced would take care of the imbalanced data $\endgroup$
    – skrrrt
    Commented Jun 1, 2020 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.