0
$\begingroup$

First, please forgive my ignorance; I am a newbie but dedicated to learning more.

Example: I have a using a random forest classifier to predict a binary outcome. The binary outcome equals 1 if people in the dataset ever experience a specific health condition and equals 0 if they don't experience the health condition.

I have tuned and run the model using the scikit-learn and related packages in a Python coding environment. I then produce the following classification report pasted below. I understand how to interpret accuracy, precision, recall (sensitivity), etc. But I am confused about the following...

Question: I want to report the most appropriate performance metrics from the classification_report -- whether just the metrics for the "= 1" class or the macro averages for both classes (ie, 0 and 1). So, if I have a binary outcome, and = 1 (ie, Yes) is the outcome of interest for prediction purposes, would it be more appropriate to report and discuss the precision, recall, F1 for the "= 1" class? Or would it instead be more appropriate and useful to report macro (or weighted) average metrics for precision, recall, F1, taking into account the metrics for both classes? For instance, I understand that the "= 0" class precision and recall metrics are better in this example, and those would affect the macro/weighted average metrics in the classification report. But I am not quite sure if it would be appropriate (and useful) to report the averages across both classes (0 and 1) in a table, rather than just the "= 1" class performance metrics. For example, macro average precision = 0.715 instead of 0.494 for just the = 1 class.

Thank you very much in advance for your time, understanding, and help!

enter image description here

Improve this question
$\endgroup$
1
  • $\begingroup$ Those metrics all require a cutoff value. A cutoff value is some value of the model score to predict a 0 vs 1 (true positive, FP, TN, FN). Selecting the appropriate cutoff value is another step that often weighs the cost/benefit of TP, FP, FN, TN. Those costs are associated with the business problem. You may want to look elsewhere for initial metrics based on the model, like lift and gain charts. $\endgroup$
    – Craig
    Jan 31, 2022 at 18:11

1 Answer 1

Reset to default
0
$\begingroup$

For your case it seems like an imbalanced class problem as the ratio of 0:1 is around 5:1. In this case, I would look at the individual score at 0 and 1. We should give a lot of importance to 1 metrics in this case.

Also, it is always good to know what your business objective is and align your evaluation metric with that.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you - yes, it is imbalanced. The = 1 class if relatively rare. $\endgroup$
    – iamtheonewhoknocks
    Jan 31, 2022 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.