0
$\begingroup$

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set and then split it to train and validate set or first split into train and val set and then perform sampling(over/under) on the train set only

$\endgroup$
1
$\begingroup$

Oversample the train data and NOT the validation data since if train data is unbalanced, your test data will most likely show the same trait and be unbalanced.

If you don't know if test data will be balanced or not, oversample only train data.

But there is one recommendation which I will personally give you since I suffered from same problem not so long ago.

Oversample the data (train) Test accuracy on validation data (which is not oversampled) Test this accuracy with accuracy obtained from not doing oversampling (or undersampling whichever you performed)

If the results vary only marginally, train the model on non oversampled data. The reason being since the oversampling technique will introduce data points near current data points belonging to same class which may not accurately depict your test data.

In such a case there are two things you can do

  1. Tweak the model parameters if available

For example : if using random forest it has a parameter in model called "class_weight" which if kept at "balanced" will give equal weightage to every output variable which would be inversely proportional to class frequencies in input

Similar parameter is available on other models as well (logistic regression etc.)

  1. When splitting data using train_test_split set parameter stratify

Example : train_test_split(train_data, df['target_column'], stratify = df['target_column'])

Stratify will make sure your train and validation data are split based on output label frequencies based on train data. Like if the data was like 90 to class 'A' and 10 to class 'B'. After split both train and validation will have 90:10 ratio of classes

$\endgroup$
0
$\begingroup$

If your future data also comes with imbalance(after deploying), don't balance the Validation set, only do oversample/undersample in the train set only. If you balance the Validation set, your model may work well(may get better score in Val) but in the future after deploying, it may not work better so while training, validate with imbalance data only.

$\endgroup$
  • $\begingroup$ Since my test set also suffers with class imbalance, I am not sampling the val set and my reason is same as yours. But I couldn't find relevant literature to support it. Do you have any paper or blog advocating this approach? $\endgroup$ – yamini goel Oct 17 at 12:42
  • $\begingroup$ researchgate.net/publication/…. This may give some analysis on that. $\endgroup$ – Uday Oct 17 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.