I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.

As I see, the correct way of doing this is to sample ONLY the train set, in order to have a test performance that is more similar to reality. Moreover, I want to use CV for hyperparameters tuning. So, the tasks in order are

  1. Divide dataset into train-test
  2. Perform the 5 fold-CV, as...
  3. Sample the "training" portion of the CV
  4. Sample the "validating" portion of the CV
  5. Train the model on the "training"
  6. Validate it on the "validating"
  7. Repeat 3-6 5 times
  8. Evaluate performances on test

My doubt is: how can I compare the CV performances with the test, since the formare are based on sampled data and the latter does not?

An idea is to skip 4 and sample only "training" portion, but in this case how can I compare the "training" with the "validating"?

EDIT: added target ratio.

  • $\begingroup$ Can you update the question with your class ratio's? Thnx. $\endgroup$ – Aditya Nov 4 '19 at 12:12
  • $\begingroup$ Sure! Question updated $\endgroup$ – Matteo Felici Nov 4 '19 at 13:39
  • $\begingroup$ If you've settled on oversampling as your balancing strategy, then why not oversample the "train" split before steps 2-7? Split the dataset into train-test splits. Apply sampling to the train split, but leave the test split alone. Proceed with CV parameter tuning as if you have a balanced dataset. $\endgroup$ – zachdj Nov 4 '19 at 17:10
  • $\begingroup$ Because, if I oversample before 2, I basically copy-paste some "1-target" observations. Then, when I do CV, I could potentially have the same record both in the "training" portion and in the "validating" portion. $\endgroup$ – Matteo Felici Nov 5 '19 at 13:46
  • $\begingroup$ Which metrics are you using for validation? $\endgroup$ – Piotr Rarus Dec 5 '19 at 8:56
I believe, the sequence for combination of CV and SMOTE should be as below.

1. Perform the 5 fold-CV ( Loop through for each fold )
2. Training Sample and Testing Sample ( for each fold )
3. Smote Training Samples
4. Train the model on the "training"
5. Prediction ( test samples )
6. Evaluate performances on test
Repeat for next fold
  • $\begingroup$ Please refer below link stackoverflow.com/questions/55591063/… $\endgroup$ – SUN Nov 5 '19 at 1:39
  • $\begingroup$ With your solution I basically skip the validation of a test-set, using only CV. By the way, it does not solve my issue: with these 6 steps, I valuate on test set that is not sampled, so I cannot compare performances on train and on test, so I cannot know if I'm overfitting. $\endgroup$ – Matteo Felici Nov 5 '19 at 13:48
  • $\begingroup$ I didn't get what do you mean by "I valuate on test set that is not sampled". May be I didn't get your question, My understanding is you want to build the prediction model which is combination of K-fold and Smote. SMOTE to overcome the problem of imbalanced data sets. When you perform SMOTE before splitting, then there is always the possibility of duplicate examples in Train and Test. So we have identify 5 sets and for each iteration, 4 sets consider as Train and 1 sets as Validation. Applied Smote on train and validate against Validation sets for the performance of Model. $\endgroup$ – SUN Nov 5 '19 at 15:52
  • $\begingroup$ In fact, I apply SMOTE after the split, not before. My question is on which datasets I should apply it. If I apply it on one ds and not on another, I cannot compare the performances coming from these two ds. In your example, I cannot compare the performances of train set (calculated between points 4 and 5) and performances on test (point 6); if I cannot compare train performances with anything, how can I know if I'm overfitting? $\endgroup$ – Matteo Felici Nov 6 '19 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.