7
$\begingroup$

I am dealing with a highly unbalanced data, so I used the SMOTE algorithm to resample the dataset.

After SMOTE resampling, I splitted the resampled dataset to training/testing sets, using the training set to build a model and testing set to evaluate the model.

However, I am worried about that some data points in the testing sets might actually jittered from data points in the training set (i.e. the information is leaking from the training set to testing set), so the testing set is not really a clean set for testing.

Does anyone have similar experience? Does the information really leak from training to testing? Or the SMOTE algorithm actually took care of it and we don't have to worry about it?

Thanks a lot!

$\endgroup$
11
$\begingroup$

When use any sampling technique ( specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you train you use the testing set ( which contains only original samples) to evaluate. The risk if you use your strategy is to have the original sample in training ( testing) and the synthetic sample ( that was created based on this original sample) in the testing ( training) set.

$\endgroup$
  • 1
    $\begingroup$ Thanks a lot, certainly understand your point. Then I am wondering this way, I won't be able to perform n-fold cross validation, right? Because my data is so small (especially for the minority class), I am trying to use as many of them as possible. $\endgroup$ – Edamame Dec 9 '16 at 18:26
2
$\begingroup$

Per your last question :

Then I am wondering this way, I won't be able to perform n-fold cross validation, right? Because my data is so small (especially for the minority class)

This is not true. You can try upsampling if your data is really small (but how small is it?)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.