2
$\begingroup$

I am working on a binary classification problem with class proportion of 33:67.

Currently what I am doing is running multiple models like LR,SVM,RF,XgBoost for classification.

RF and Xgboost models perform better.

However I am reading online that AUC, F1-score or Accuracy comparison between models may not be a good metric (as they are sensitive) to measure the model performance.

I see these are the mostly used scoring metrics but can you let me know which metric should I look for measuring binary classification model performance?

Can you let me know why should we choose a different metric?

These are the metrics that I see in scikit-learn

enter image description here

$\endgroup$
1
$\begingroup$

F1 is just based on the confusion matrix(and taking into account class imbalance), hence different models should only focus on predicting the confusion matrix correctly and if they dont they are wrong not sensitive.

F1Score is a metric to evaluate predictors performance using the formula

F1 = 2 * (precision * recall) / (precision + recall)

recall = TP/(TP+FN)

precision = TP/(TP+FP)

but the main thing is:

enter image description here

$\endgroup$
  • $\begingroup$ Thanks for the response. Upvoted. So you are suggesting that we should pick the model which performs best work of predicting the confusion matrix. Right? $\endgroup$ – The Great Jan 4 at 11:21
  • $\begingroup$ I was referring to basis of derivation for f1 score. I am suggesting stick with f1 for most cases. I am sure that you can construct some edge examples where it fails but f1 is stable 99% of the time and if the model scores bad it is bad $\endgroup$ – Noah Weber Jan 4 at 11:22
  • $\begingroup$ @NoahWeber do you have some references for that "99% of the time" stats? $\endgroup$ – BrunoGL Jan 4 at 12:16
  • $\begingroup$ Never had a single personal case, its possible. As a mathematician I say don't trust the theory. Naive bayes shouldn't work but it works $\endgroup$ – Noah Weber Jan 4 at 12:23
2
$\begingroup$

TL;DR

I would suggest you to:

  1. Balance your dataset (since you are not doing an anomaly detection task, that should be fine).
  2. Measure Precision/Recall/F1 Score and argue for them depending on your problem.
  3. Perform K-Fold Cross Validation to compare models

Balancing your dataset

You said you have a 33:67 ratio. Why keep it like that an not just make it 50:50 (e.g. by removing samples from the larger class)? In general this help your models be less biased to the larger class.

Special care is needed when doing anomaly detection, where ratios are usually around 1:99. But since you are not doing that, simple dataset balancing is advisable.

Precision / Recall / F1

F1 score is the default score to go to for classification. It should be ok to use across models as long as you keep your test set the same for all models. An even better thing to do to compare models, is to perform K-Fold Cross Validation on each model. (Sklean docs).

However, you could look at Precision or Recall instead of the F1 Score, but this will depend on the problem you are trying to solve.

Precision = when I detect an instance of a class, I am sure it is the right class.

Recall = I detect all instances of a class, even if I detect instances that don't belong.

For example: we are performing a binary classification of whether a patient has a deadly disease or not. Health Insurance: "let's optimize the precision, because we only want to pay treatment for patients that really have the disease". Doctor: "let's optimize for recall, because it is better to treat more people even if the don't have the disease to ensure that those that do have it are likely being taken care of".

If you think that precision are recall are equally important then that is the definition of the F1 score.

$\endgroup$
  • $\begingroup$ Hi, thanks for the response. Upvoted $\endgroup$ – The Great Jan 6 at 23:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.