unbalanced data on train set and test set

Question

I already have 2 datasets. One to use for training and one for testing. Both datasets are unbalanced (with similar percentages), with around 90% of label 1 . Will it be useful to balance the data if the test set is very unbalanced anyway? Instances of label 0 (which are 10%) are still enough.

If necessary, I would eventually use oversampling. Mine is a tripAdvisor review dataset, what would be the best technique to use in this case?

Are there any mixed techniques that use both undersampling and oversampling? or does it make little sense?

Answer 1

If your dataset is sufficiently large and you might want to reduce its size for performance reasons anyways, you could do undersampling of label 1.

However, if you only have a limited amount of data available or if you definitely need all training samples to be able to build a stable classifier, I'd go with oversampling (Random Oversampling (which duplicates data points from label 0), Synthetic Minority Oversampling Technique SMOTE (which focuses specifically on samples close to the border between label 0 and 1) or Adaptive Synthetic Sampling ADASYN (which focuses on hard-to-classify minority classes)).

Personally, SMOTE worked well for many imbalanced datasets, give it a try! It helps your classifier differentiating better between classes and can thus improve performance even if the test set is equally imbalanced.

But remember to only use it on the training data to be able to do an unbiased evaluation using the test set.

Answer 2

It's also possible to decrease the learning step when updating weights learned from the majority class, and/or increase the learning step when updating weights learned from the minority class.

See class-weights in Scipy.

Answer 3

It is often useful to balance a training dataset. For example, if the model learns a decision boundary, that decision boundary will then learn to separate different categories based more on features than the proportion of observations from the majority classes. During the prediction-only phase on a testing dataset, that same decision boundary is used with no rebalancing.

Undersampling and oversampling are independent. Oversampling is sampling with replacement from the minority classes. Undersampling is downsampling the majority classes. You can do both or either one.

The goal is to have a good representation of the data while minimizing training time. If you have many, high-quality data points, do more undersampling. If you have few, low-quality data points, do more oversampling.