1
$\begingroup$

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.

As I see, the correct way of doing this is to sample ONLY the train set, in order to have a test performance that is more similar to reality. Moreover, I want to use CV for hyperparameters tuning. So, the tasks in order are

  1. Divide dataset into train-test
  2. Perform the 5 fold-CV, as...
  3. Sample the "training" portion of the CV
  4. Sample the "validating" portion of the CV
  5. Train the model on the "training"
  6. Validate it on the "validating"
  7. Repeat 3-6 5 times
  8. Evaluate performances on test

My doubt is: how can I compare the CV performances with the test, since the former are based on sampled data and the latter does not?

An idea is to skip 4 and sample only "training" portion, but in this case how can I compare the "training" with the "validating"?

EDIT: added target ratio + typo.

$\endgroup$
9
  • $\begingroup$ Can you update the question with your class ratio's? Thnx. $\endgroup$
    – Aditya
    Nov 4, 2019 at 12:12
  • $\begingroup$ Sure! Question updated $\endgroup$ Nov 4, 2019 at 13:39
  • $\begingroup$ If you've settled on oversampling as your balancing strategy, then why not oversample the "train" split before steps 2-7? Split the dataset into train-test splits. Apply sampling to the train split, but leave the test split alone. Proceed with CV parameter tuning as if you have a balanced dataset. $\endgroup$
    – zachdj
    Nov 4, 2019 at 17:10
  • 1
    $\begingroup$ Because, if I oversample before 2, I basically copy-paste some "1-target" observations. Then, when I do CV, I could potentially have the same record both in the "training" portion and in the "validating" portion. $\endgroup$ Nov 5, 2019 at 13:46
  • $\begingroup$ Which metrics are you using for validation? $\endgroup$ Dec 5, 2019 at 8:56

2 Answers 2

1
$\begingroup$

The first issue is why do you need to use SMOTE? Imbalanced datasets, provided they are sufficiently large, do not present a significant problem for statistical classifiers or machine learning methods. If you have an imbalanced dataset, quite often the optimal accuracy is obtained by assigning everything to the majority class. If that is not acceptable, it is an indication that the minority class is more "important" in some sense than the majority class (i.e. there ought to be a higher cost for missclassifying a minority class pattern as belonging to the majority class than vice versa). In other words, accuracy is not the right performance metric and you need to look at the expected loss (effectively a weighted accuracy - weighted according to the misclassification costs). So rather than using SMOTE, it would be better to see if you can work out what the misclassification costs actually should be and incorporate those into the classifier (either by changing the threshold probability for a probabilistic classifier, or by weighting the positive and negative patterns unequally in the training criterion). Most often "class imbalance problems" are just "cost sensitive learning problems" in disguise.

Note that SMOTE was originally developed in the context of very primitive classifier systems, such as single decision trees or RIPPER, that were prone to over-fitting if minority examples were simply resampled. The generation of synthetic examples acts to "blur" the minority examples, so they are more difficult to overfit. Modern classifier systems have effective means of avoiding overfitting, such as regularisation, so it is questionable whether the rather odd way in which SMOTE generates synthetic examples is a good idea for modern methods.

If you are tuning hyper-parameters to optimise operational peformance, then the "test" folds in cross-validation should be representative of operational conditions, so you should not be applying SMOTE to them or resampling, if your original dataset was representative of operational conditions.

$\endgroup$
0
$\begingroup$
I believe, the sequence for combination of CV and SMOTE should be as below.

1. Perform the 5 fold-CV ( Loop through for each fold )
2. Training Sample and Testing Sample ( for each fold )
3. Smote Training Samples
4. Train the model on the "training"
5. Prediction ( test samples )
6. Evaluate performances on test
Repeat for next fold
$\endgroup$
4
  • $\begingroup$ Please refer below link stackoverflow.com/questions/55591063/… $\endgroup$
    – SUN
    Nov 5, 2019 at 1:39
  • $\begingroup$ With your solution I basically skip the validation of a test-set, using only CV. By the way, it does not solve my issue: with these 6 steps, I valuate on test set that is not sampled, so I cannot compare performances on train and on test, so I cannot know if I'm overfitting. $\endgroup$ Nov 5, 2019 at 13:48
  • $\begingroup$ I didn't get what do you mean by "I valuate on test set that is not sampled". May be I didn't get your question, My understanding is you want to build the prediction model which is combination of K-fold and Smote. SMOTE to overcome the problem of imbalanced data sets. When you perform SMOTE before splitting, then there is always the possibility of duplicate examples in Train and Test. So we have identify 5 sets and for each iteration, 4 sets consider as Train and 1 sets as Validation. Applied Smote on train and validate against Validation sets for the performance of Model. $\endgroup$
    – SUN
    Nov 5, 2019 at 15:52
  • $\begingroup$ In fact, I apply SMOTE after the split, not before. My question is on which datasets I should apply it. If I apply it on one ds and not on another, I cannot compare the performances coming from these two ds. In your example, I cannot compare the performances of train set (calculated between points 4 and 5) and performances on test (point 6); if I cannot compare train performances with anything, how can I know if I'm overfitting? $\endgroup$ Nov 6, 2019 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.