Oversampling/Undersampling only train set only or both train and validation set

Question

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set and then split it to train and validate set or first split into train and val set and then perform sampling(over/under) on the train set only

Answer 1

Oversample the train data and NOT the validation data since if train data is unbalanced, your test data will most likely show the same trait and be unbalanced.

If you don't know if test data will be balanced or not, oversample only train data.

But there is one recommendation which I will personally give you since I suffered from same problem not so long ago.

Oversample the data (train) Test accuracy on validation data (which is not oversampled) Test this accuracy with accuracy obtained from not doing oversampling (or undersampling whichever you performed)

If the results vary only marginally, train the model on non oversampled data. The reason being since the oversampling technique will introduce data points near current data points belonging to same class which may not accurately depict your test data.

In such a case there are two things you can do

1. Tweak the model parameters if available

For example : if using random forest it has a parameter in model called "class_weight" which if kept at "balanced" will give equal weightage to every output variable which would be inversely proportional to class frequencies in input

Similar parameter is available on other models as well (logistic regression etc.)

2. When splitting data using train_test_split set parameter stratify

Example : train_test_split(train_data, df['target_column'], stratify = df['target_column'])

Stratify will make sure your train and validation data are split based on output label frequencies based on train data. Like if the data was like 90 to class 'A' and 10 to class 'B'. After split both train and validation will have 90:10 ratio of classes

Answer 2

If your future data also comes with imbalance(after deploying), don't balance the Validation set, only do oversample/undersample in the train set only. If you balance the Validation set, your model may work well(may get better score in Val) but in the future after deploying, it may not work better so while training, validate with imbalance data only.