2
$\begingroup$

it is my first time doing something with financial data. I have a dataset with account numbers and some other information about each client (some clients span more than one row since we have info for each month in a different row). I managed to clean and create some models, here are the confusion matrices, the classification reports and AUC:

Logistic regression

[[185847  62897]
 [     1   1061]]


             precision    recall  f1-score   support

     not buy       1.00      0.75      0.86    248744
         buy       0.02      1.00      0.03      1062

    accuracy                           0.75    249806
   macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.75      0.85    249806

AUC train = 0.9168592981611143
AUC test = 0.9150300677458543

Random Forest Classifier:

[[245503   3241]
 [   960    102]]


              precision    recall  f1-score   support

     not buy       1.00      0.99      0.99    248744
         buy       0.03      0.10      0.05      1062

    accuracy                           0.98    249806
   macro avg       0.51      0.54      0.52    249806
weighted avg       0.99      0.98      0.99    249806

AUC train = 0.9996866568080237
AUC test = 0.9139101966925902

Gradient Boosting Classifier:

[[184940  63804]
 [     3   1059]]


              precision    recall  f1-score   support

     not buy       1.00      0.74      0.85    248744
         buy       0.02      1.00      0.03      1062

    accuracy                           0.74    249806
   macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.74      0.85    249806

AUC train = 0.8800353734759541
AUC test = 0.8657829269466372

Voting Classifier (from all the three above):

[[211316  37428]
 [   213    849]]


              precision    recall  f1-score   support

     not buy       1.00      0.85      0.92    248744
         buy       0.02      0.80      0.04      1062

    accuracy                           0.85    249806
   macro avg       0.51      0.82      0.48    249806
weighted avg       0.99      0.85      0.91    249806

AUC train = 0.9987531510931085
AUC test = 0.9160262741936392

Since I do not have any experience I am not sure which model is producing better results. Can you help me understand which one and why? Thank you!

$\endgroup$

2 Answers 2

2
$\begingroup$

Well, evaluating your model just by comparing matrixes can be pretty hard. In your case, your matrixes are not done with the same number of row classified 1 or 0, so comparing them is really hard. Let's take an example : your logistic regression classifies around 64000 as 1, while your RandomForest only classifies around 4500, so comparing these data by this matrix is pretty hard.

I'd suggest you to use the ROC AUC metric, that is really useful to compare models. You might find many info on this subject in the Internet. The closer your AUC is than 1, the better your model is. If it's 0.5 or less, your model is less efficient than a random classifier.

$\endgroup$
8
  • $\begingroup$ What you can do if you REALLY want to use matrixes as a comparing tool is tweaking the threshold to have the same amount of rows classified 1 and 0 on each of your model. With this proportion you can then look at the FP and FN, and compare your models $\endgroup$
    – Adept
    Aug 11, 2020 at 7:58
  • 1
    $\begingroup$ thank you for your comment and answer. I did calculate also ROC AUC for each one. I have: 0.92, 0.91, 0.87 and 0.91 respectively in the same order as above. I will post also the classification reports now. $\endgroup$ Aug 11, 2020 at 8:05
  • 1
    $\begingroup$ Yes I was using GridSearchCV and tried several parameters for each mode, so now they are fitted with the best parameters. Based on AUC you would go with Logistic regression right? (I have a small reputation and I can't upvote your comments to show my gratitude) $\endgroup$ Aug 11, 2020 at 8:20
  • 1
    $\begingroup$ It looks like Random Forest and Voting classifiers are overfitting. I will have to tune them. $\endgroup$ Aug 11, 2020 at 9:08
  • 1
    $\begingroup$ Okay I managed to tune it a bit: AUC train : 0.9634409585009942, AUC test: 0.9129260167677515. Thanks for the help! $\endgroup$ Aug 11, 2020 at 9:34
2
$\begingroup$

There are some different methods you can use to measure your model's accuracy. As @BeamsAdept mentioned in his own answer you can use the ROC AUC metric. Alternatively you mind find the Odds Ratio useful and if your dataset is big enough I would highly encourage you look into something like K-fold cross validation in order to get more representative results. I would also be cautious when the training accuracy is significantly higher than your testing accuracy which is the result of overfitting. Perhaps finding the P-values of your features would help you filter out potentially useless information and as a result improve your accuracy in new observations. Hope this helps!

$\endgroup$
2
  • $\begingroup$ You're totally right, I forgot to mention Cross Validation and using the mean of your results to have a more precise one $\endgroup$
    – Adept
    Aug 11, 2020 at 13:53
  • $\begingroup$ Hi, thank you for your answer! I was using GridSearchCV (k=5) for all the models that I tried. Unfortunately I didn't do a feature selection for logistic regression, and for RandomForest and GradientBoosting I think it is done automatically. I will try to understand Odds Ratio bettr because for now it confuses me. $\endgroup$ Aug 12, 2020 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.