8
$\begingroup$

I have a dataset with 4519 samples labeled as "1", and 18921 samples labeled as "0" in a binary classification exercise. I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class.

However, should the test dataset be balanced as well?

In other words, if train my model with 1000 random samples of "0" class, and 1000 random samples of "1" class, should I test the model with the remaining 3519 samples of "1" class, and randomly select another 3519 samples of the majority "0" class, or I can go with the remaining 17921?

What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Thanks

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ In case of neural networks, I would would train the algorithm $\frac{18921}{4519}$ more vigorously on class "1" (as a learning rate multiplier), and would do a similar trick during test time. Basically Importance Sampling. Here is a good video about it: youtube.com/watch?v=ymeSc0SPFmE $\endgroup$
    – Kari
    Commented Nov 29, 2018 at 12:52
  • 2
    $\begingroup$ "I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class." This is absolutely false. Models that predict probabilities make no such assumptions. See here: stats.stackexchange.com/questions/283170/… and stats.stackexchange.com/questions/312780/… $\endgroup$
    – Matthew Drury
    Commented Nov 29, 2018 at 18:36
  • $\begingroup$ Thanks for pointing at these two discussions, I didn't know then and there are useful insights. However, data imbalance is indeed a problem for machine learning algorithms, please see: link.springer.com/content/pdf/10.1007%2Fs13748-016-0094-0.pdf. The effect could be less on probabilistic algorithms (not gonna argue that), but Random Forest is an ensemble learning algorithm, so it has no built-in capacity for probability or to deal with imbalanced distributions (and returning the probability of belonging to a class, is not really working with "probabilties"...). Thanks anyway. $\endgroup$
    – Irene
    Commented Dec 12, 2018 at 8:46
  • $\begingroup$ I dispute the idea that class imbalance is a problem. I will give some other links, in addition to those posted by @MatthewDrury: 1 2 3. $\endgroup$
    – Dave
    Commented Aug 26, 2021 at 13:01

5 Answers 5

Reset to default
7
$\begingroup$

The answer to your first question:

should the test dataset be balanced as well?

is, like many answers in data science, "it depends."

And really, it depends on the audience for, and interpretability of the model metrics, which is the thrust of your second question:

What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Personally, if the metrics will just be used by you to evaluate the model, I would use the sensitivity and specificity within each class to evaluate the model, in which case, I care less about the balance of the classes in the test data as long as I have enough of both to be representative. I can account for the prior probabilities of the classes to evaluate the performance of the model.

On the other hand, if the metrics will be used to describe predictive power to a non-technical audience, say upper management, I would want to be able to discuss the overall accuracy, for which, I would want a reasonably balanced test set.

That said, it sounds like your test set is drawn independently of the training data. If you are going to balance the training data set, why not draw one balanced data set from the raw data and then split the training and test data? This will give you very similar class populations in both data sets without necessarily having to do any extra work.

Improve this answer
$\endgroup$
1
$\begingroup$

There are several ways of balancing classes. Either you can increase the number of samples of minority class or decrease the number of samples of majority class. Once you have balanced your classes, for a start you take 80% of it as train and test it on remaining 20%. In your case 4519 samples are 1 and 18921 are 0 so lets say you upsample 4519 and now you have 15,000 samples of class 1. You will take 80% of (15000+18921) and train your model. Test it on remaining 20%.

Recall becomes important when you are let's say identifying fraudulent transactions out of several millions. You will have very few classes labelled as 1 (fraudulent). So even you simply label all the classes in your test data as 0 you will be more than 99% accurate but you should ask yourself the question "Out of all the actual classes with label 1 how many did i get right from the trained model". That is what recall gives you.

Improve this answer
$\endgroup$
1
$\begingroup$

Your model could have very dissimilar performances at detecting "0" or "1" samples without you noticing if the performance is not high enough.

I would include more samples in the training and subsample "0" (to keep balance of "0" and "1" in each step) during the training and would evaluate the model with a balanced test set.

Improve this answer
$\endgroup$
1
$\begingroup$

A fast solution: If you use SKLearn's Random Forest, I highly suggest "class_weight" => “balanced”.

The argument will automatically weigh classes inversely proportional to their frequency.

Improve this answer
$\endgroup$
1
$\begingroup$

I don't think test dataset should follow the ratio of classes of training one. Class ratio just follows that of real population s.t. test result properly reflects what'll actually happen with the actual predictions in the real world. Class balancing/sampling in training phase with training dataset is independently optimized to the evaluation performance over the test dataset.

And the class weight techniques become useful in case the classes are so imbalanced in your test dataset, to avoid any confusion matrix paradox.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.