3
$\begingroup$

I would like to ask you some questions on how to consider (good or not) the following results:

OVER-SAMPLING
              precision    recall  f1-score   support

         0.0       1.00      0.85      0.92       873
         1.0       0.87      1.00      0.93       884

    accuracy                           0.92      1757
   macro avg       0.93      0.92      0.92      1757
weighted avg       0.93      0.92      0.92      1757

Confusion Matrix: 
 [[742 131]
 [  2 882]]

I have a dataset with 3500 obs (3000 with class 0 and 500 with class 1). I would like to predict class 1 (target variable). Since it is a problem of imbalance classes, I had to consider re-sampling methods. The result shown above is from over-sampling. Do you think it over-fits and/or that it cannot be a good re-sampling method for my case? I am looking at the f1-score column, since it is a text classification problem.

Improve this question
$\endgroup$
4
  • $\begingroup$ Highly relevant: stats.stackexchange.com/questions/247871/… $\endgroup$
    – cbeleites
    Commented Nov 30, 2020 at 14:47
  • $\begingroup$ thanks @cbeleitesunhappywithSX. I think my main difficulties is in understand differences in running before and after the training dataset. I did it using SMOTE function, and I found an acceptable value for recall (0.71 for class 0 and 0,87 for class 1) but not for accuracy, precision and f1. Do you think it is a problem of overfitting? $\endgroup$
    – V_sqrt
    Commented Nov 30, 2020 at 14:50
  • $\begingroup$ I think the problems lie deeper, likely in a wrong ansatz in general and for sure in your incomplete understanding in how to split data into independent train and test subsets. The latter also causes additional problems: with your code you are unable to detect all of the overfitting, and many figures of merit ("kinds" of generalization error) you calculate from the oversampled test set will be irrelevant for the application task. Likely also the model you train on the oversampled data is not so very good for the application, but with proper validation you'd at least know that. $\endgroup$
    – cbeleites
    Commented Nov 30, 2020 at 15:22
  • $\begingroup$ Please go through datascience.stackexchange.com/questions/105125/… $\endgroup$
    – Encipher
    Commented Dec 14, 2021 at 22:10

2 Answers 2

Reset to default
5
$\begingroup$

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification report should mirror the imbalance in your dataset.

From what I understand you have 3500 samples, then you did some oversampling (probably brought them to around 6000) and then took 1757 from these for testing. This evaluation scheme is wrong. Take a look at the illustration below to see a more correct scheme.

      |--- train --> oversample train set --> train model---|
set --|                                                     |--> evaluation on test set
      |--- test --------------------------------------------|
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Djib2011. I updated the question in order to show the steps I followed. could you please have a look at tell me if you spot any inconsistencies with what you suggested? I think I already upsampled only the training set, but I would like to be sure I have not made any mistake. Many thanks. $\endgroup$
    – V_sqrt
    Commented Nov 30, 2020 at 11:25
  • 2
    $\begingroup$ @Val since you first upsample to X_up and you use train_test_split on this upsampled set, you are indeed committing the mistake; the correct sequence is split first, then upsample only the training set. See Why you shouldn't upsample before cross validation for the rationale (it's about CV, but it is also applicable to train/test splitting). $\endgroup$
    – desertnaut
    Commented Nov 30, 2020 at 12:36
  • $\begingroup$ Thanks a lot, desertnaut. So I will change the sequence (first split, then upsample). Unfortunately, I followed the steps from TowardsDataScience/Medium and I made the mistake. Thanks a lot $\endgroup$
    – V_sqrt
    Commented Nov 30, 2020 at 13:01
2
$\begingroup$

In order to detect overfitting you need to separate your data in a training set - that you use to estimate the parameters of you model - and a test set - where you evaluate your model keeping the parameters fixed, this is usually called cross-validation. I understand from your results that you are not doing such separation of data so, you're not able to detect overfitting.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi andis, I got these values from cross_valid in python: array([0.95121951, 0.96341463, 0.95609756, 0.93902439, 0.95609756, 0.95365854, 0.96097561, 0.94621027, 0.95110024, 0.97066015]). Are you sure that cross-validation works? Have you spotted any inconsistency in my code? $\endgroup$
    – V_sqrt
    Commented Nov 30, 2020 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.