21
$\begingroup$

I have a model that does binary classification.

My dataset is highly unbalanced, so I thought that I should balance it by undersampling before I train the model. So balance the dataset and then split it randomly. Is this the right way ? or should I balance also the test and train dataset ?

I tried balancing only the whole dataset and I get train accuracy of 80% but then on the test set I have 30% accuracy. This doesn't seem right ?

But also I don't think that I should balance the test set because it could be considered as bias.

What is the right way to do this?

Thanks

UPDATE: I have 400 000 samples, 10% are 1s and 90% 0s. I cannot get more data. I tried to keep the whole dataset but I don't know how to split it into train and test set. Do I need the same distribution in the train and test dataset ?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ How much data you have? and what algorithms are you using for it? Rather than under sampling or oversampling, try doing resampling. Also, use other metrics than accuracy, like Precision, recall or F1-score. $\endgroup$
    – GadaaDhaariGeek
    Jun 8, 2018 at 10:11

4 Answers 4

Reset to default
24
$\begingroup$

Best way is to collect more data, if you can.

Sampling should always be done on train dataset. If you are using python, scikit-learn has some really cool packages to help you with this. Random sampling is a very bad option for splitting. Try stratified sampling. This splits your class proportionally between training and test set.

Run oversampling, undersampling or hybrid techniques on training set. Again, if you are using scikit-learn and logistic regression, there's a parameter called class-weight. Set this to balanced.

Selection of evaluation metric also plays a very important role in model selection. Accuracy never helps in imbalanced dataset. Try, Area under ROC or precision and recall depending on your need. Do you want to give more weightage to false positive rate or false negative rate?

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the answer. Just to confirm, I will do under sampling so I will have 50% 0s and 50% 1s, then I will split proportionaly into train and test set(meaning that I will have 50:50 0s and 1s in the train and test set). Then I will use F1-score and/or precision and recall ? Is it ok that I know the distribution in the test set beforehand ? $\endgroup$
    – lads
    Jun 8, 2018 at 11:15
  • 5
    $\begingroup$ No, split into training and test set first. Perform sampling technique on training set alone. $\endgroup$
    – aathiraks
    Jun 8, 2018 at 12:00
  • $\begingroup$ I have one more question, I did as you said, but after oversampling the train set I get accuracy, recall, precision all around 0.5. Do you know what could be the reason for this ? I tried using LogisticRegression from sklearn and also create my own model with linear layer and sigmoid activation function, but i get the same result. $\endgroup$
    – lads
    Jun 8, 2018 at 12:44
  • $\begingroup$ Model isn't good enough. Try logistic regression with class_weight as balanced without sampling. Also, try boosting techniques. Use GridSearchCV to find the best values for parameters. $\endgroup$
    – aathiraks
    Jun 8, 2018 at 13:04
  • $\begingroup$ So when splitting original datasets into train and test, we should use stratified sampling not simple random sampling, right? $\endgroup$
    – Spaceship222
    Sep 18, 2020 at 4:43
6
$\begingroup$

You problem is very common and many data scientists are struggling with there kind of issues.

In this blog post, the author explain very nicely what to do. Those are the main notes:

1. Can You Collect More Data?

2. Try Changing Your Performance Metric:

Accuracy is not the metric to use when working with an imbalanced dataset. We have seen that it is misleading.

There are metrics that have been designed to tell you a more truthful story when working with imbalanced classes.

Precision: A measure of a classifiers exactness. Recall: A measure of a classifiers completeness F1 Score (or F-score): A weighted average of precision and recall.

3. Resampling Your Dataset

You can change the dataset that you use to build your predictive model to have more balanced data.

This change is called sampling your dataset and there are two main methods that you can use to even-up the classes:

  • You can add copies of instances from the under-represented class called over-sampling (or more formally sampling with replacement), or

  • You can delete instances from the over-represented class, called under-sampling.

4. Generate Synthetic Samples

simple way to generate synthetic samples is to randomly sample the attributes from instances in the minority class.

5. Try Different Algorithms

As always, I strongly advice you to not use your favorite algorithm on every problem. You should at least be spot-checking a variety of different types of algorithms on a given problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ hi Gal, I cannot correct more data. I tried training with the imbalance dataset and I got recall 0.99 and precision 0.9. I tested on a dataset that follows the same distribution and I got f1-score=0.94 same sa in the training. This is not normal right ? I decided to keep the whole imbalance dataset (400 000 samples) and use F1-score as metric, but I don't know how to spit it into test and train ? My question is do the test and train dataset need to follow the same distribution of 0s and 1s ? $\endgroup$
    – lads
    Jun 8, 2018 at 10:49
  • $\begingroup$ In general - yes. but of course it depends on the case. My concern here is that your model may overfit the training data. Anyway, in my experience, imbalanced data may cause bad results. $\endgroup$
    – Gal Dreiman
    Jun 9, 2018 at 9:02
2
$\begingroup$

It all depends on what's your objective. Do you aim at precision or recall?

You are right the distribution of your training Data (depending always on the model and the hyper-parameters) will bias your model accordingly to it. Supplying a training set where most of the instances (i.e. 90%) are labelled as 0's, will probably label in the test set most of them as 0's. Hence, if one would like to detect the 1's should bias the sample in order to have more of these. There are many ways of doing that and beyond changing your training distribution. Firstly, oversampling, undersampling or even better, using ensemble models where each model may have all the 1s and some 0s. Secondly, one can tune depending on the classifier of choice various hyper-parameters which are responsible for constraining the majority class to take over.

Improve this answer
$\endgroup$
0
$\begingroup$

As mentioned in most of the answers that there are various ways of dealing with skewed data. I would just like to highlight that SMOTE is one of the recommended ways to overcome this skewness.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ All of the above answers covers the techniques to overcome the issue. If you choose to do upsampling/downsampling then the imblearn package in python can helpful. It includes several techniques to deal with imbalanced data in general. (I wanted to add as comment in Rahul's answer but don't have enough reputations.) $\endgroup$
    – smm
    Feb 4, 2019 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.