Timing of applying random oversampling on the dataset

Question

I tried to learn classification using machine learning algorithms. I went through Breast Cancer - EDA, Balancing and ML the notebook. In this notebook Random Oversampling had been implemented. However, when the person did the oversampling he did it on the whole dataset. I know that oversampling can be applied only to the training dataset.

In my case after splitting the data into training and test set and I applied oversampling to the training data. The precision, and recall that I have got are not as good as the Kaggle notebook.

Kaggle result

    precision    recall  f1-score   support

           0       0.73      0.90      0.81      1010
           1       0.87      0.68      0.76      1035

    accuracy                           0.79      2045
   macro avg       0.80      0.79      0.78      2045
weighted avg       0.80      0.79      0.78      2045

My result

                 precision    recall  f1-score   support

           0       0.90      0.91      0.91      1023
           1       0.49      0.46      0.47       185

    accuracy                           0.84      1208
   macro avg       0.70      0.69      0.69      1208
weighted avg       0.84      0.84      0.84      1208

This two results are for Decision tree classifier.

My code block to getting the result

from sklearn.model_selection import train_test_split
(X_train, X_test, y_train, y_test)=train_test_split(X,y,test_size=0.3, stratify=y)
from imblearn.over_sampling import RandomOverSampler
ROS = RandomOverSampler(random_state=0)
ROS_X, ROS_y = ROS.fit_resample(X_train, y_train)
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report
Random_Decision = DecisionTreeClassifier(random_state=0)
Random_Decision.fit(ROS_X, ROS_y)
D_y_pred = Random_Decision.predict(X_test)
print(classification_report(y_test, D_y_pred))

Kaggle code block

ros = RandomOverSampler(random_state=0)
X, y = ros.fit_resample(X, y)
Label encoder, max-minscaler had been used in the dataset
X_train, X_test, y_train, y_test = train_test_split(X_normalization, y, test_size = 0.3, random_state = 0)
arvore_entropy = DecisionTreeClassifier(criterion = 'entropy', max_depth= 3, random_state=0)
arvore_entropy.fit(X_train, y_train)
previsoes = arvore_entropy.predict(X_test)
classification_decision_entropy = (classification_report(y_test, previsoes))
print(classification_decision_entropy)

My code after taking the same parameter as Kaggle

Random_Decision1 = DecisionTreeClassifier(criterion = 'entropy', max_depth= 3,random_state=0)
Random_Decision1.fit(ROS_X, ROS_y)
D_y_pred1 = Random_Decision1.predict(X_test)
print(classification_report(y_test, D_y_pred1))

Output:

                  precision    recall  f1-score   support

           0       0.95      0.74      0.83      1023
           1       0.35      0.77      0.48       185

    accuracy                           0.75      1208
   macro avg       0.65      0.76      0.66      1208
weighted avg       0.86      0.75      0.78      1208

Therefore, I would like to know if am I right about applying oversampling to the training dataset.

Thank you.

Answer 1

Current answers are roughly correct, but miss the main split of when you would use it.

By default, if you are going to, only over-sample on training set. But, the key question is do you have a unbalanced dataset, or an under-represented dataset. So how well does your data reflect the real world.

If you have a unbalanced dataset, ONLY over-sample on the training set. If you have an under-represented dataset, then there are cases where it makes sense to use over-sampling techniques, such as SMOTE, on the full dataset - but must be done with extreme caution.

This all being said, I've not seen a production use case of over-sampling where it was applied correctly or outperformed under-sampling of the majority class.

Answer 2

Oversampling/Undersampling only train set only or both train and validation set

I found this answer which provides various possibilities on how to go about the random over/ under sampling.

1. Over/ undersample the train set and then validate the model using a validation set ( not over/ undersampled) and then test the model on a test set ( not over/ under sampled)

To better gauge the problem, it would be helpful to have a look at the results from your implementation

Answer 3

Let's start with the main message: It seems like you are doing it right so far.

One problem with oversampling before splitting is, that identical samples can be contained in both training and test (or all three, if you have validation sets as well). While there might be occasionally duplicates in original datasets as well, oversampling causes a much larger problem due to systematic duplicating.

Why is this a problem? With training and test sharing samples, you might not be able to spot overfitting. As a consequence, the result can look better when oversampling on the whole dataset, but this would be similar to evaluating on the training data (which also looks better then evaluation on the test set).