Which is the best binary classification model? Train and Test Accuracy are similar

Question

I am building a binary classification model where classes are imbalanced but used SMOTE, I used 4 different models to compare performance and decide which to choose. They have same train and test accuracy.When I tested the models xgboost and logistic regression only on people with credit cards,lr predicted 81% but xgboost and random forest 50% predicted right.

Model:(Train Accuracy,Test Accuracy)

1. Random Forest:(0.963,0.961)
2. Decision Tree:(0.962,0.961)
3. Logistic Regression:(0.815, 0.815)
4. XGBoost:(0.813,0.799)

Model:(roc_auc_score,accuracy_score)

• Decision Tree:(0.949,0.993)
• Logistic Regression:(0.863,0.981)
• Random Forest:(0.968,0.995)
• XGBoost:(0.964,0.995)

The images attached show the precision,recall and f1 score.

Answer 1

Unfortunately, there is not golden rule which model is the best.
It always depends on the use case and what is done with the model. In general, the metrics allow to compare different aspects, but it is up to you to decide which is more important for you use case.

You could start to consider whether a False Positive or a False Negative is worse. This would be a good starting point decide on which metric to focus. For examlple: precision only consideres False Positives, while Recall only consideres False Negatives.

Besides that, there are some guidelines:

• Focus on the evaluation on the test set. Typically, it is a better estimate of the real outcome that you could expect.
• If you use techniques like SMOTE, do not use it on the test set. Otherwise, you might be surprised be the outcome, if applying the model in real world.