-1
$\begingroup$

I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose. They have same train and test accuracy.When I tested the models xgboost and logistic regression only on people with credit cards,lr predicted 81% but xgboost and random forest 50% predicted right.

Model:(Train Accuracy,Test Accuracy)

  1. Random Forest:(0.963,0.961)
  2. Decision Tree:(0.962,0.961)
  3. Logistic Regression:(0.815, 0.815)
  4. XGBoost:(0.813,0.799)

Model:(roc_auc_score,accuracy_score)

  • Decision Tree:(0.949,0.993)
  • Logistic Regression:(0.863,0.981)
  • Random Forest:(0.968,0.995)
  • XGBoost:(0.964,0.995)

The images attached show the precision,recall and f1 score.model metrics

model classes

model classes2 here

$\endgroup$

1 Answer 1

1
$\begingroup$

Unfortunately, there is not golden rule which model is the best.
It always depends on the use case and what is done with the model. In general, the metrics allow to compare different aspects, but it is up to you to decide which is more important for you use case.

You could start to consider whether a False Positive or a False Negative is worse. This would be a good starting point decide on which metric to focus. For examlple: precision only consideres False Positives, while Recall only consideres False Negatives.

Besides that, there are some guidelines:

  • Focus on the evaluation on the test set. Typically, it is a better estimate of the real outcome that you could expect.
  • If you use techniques like SMOTE, do not use it on the test set. Otherwise, you might be surprised be the outcome, if applying the model in real world.
$\endgroup$
1
  • $\begingroup$ The "focus the evaluation on the test set" part can't be overstated. Until a model has been tested on completely unknown data, the accuracy score on the training set is pretty much useless, apart from gleaning a theoretical upper limit of what this algorithm can do for this dataset. $\endgroup$
    – liakoyras
    Jul 18, 2023 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.