0
$\begingroup$

I have a dataset with 4519 samples labeled as "1", and 18921 samples labeled as "0" in a binary classification exercise. I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class.

However, should the test dataset be balanced as well?

In other words, if train my model with 1000 random samples of "0" class, and 1000 random samples of "1" class, should I test the model with the remaining 3519 samples of "1" class, and randomly select another 3519 samples of the majority "0" class, or I can go with the remaining 17921?

What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Thanks

$\endgroup$
  • 1
    $\begingroup$ In case of neural networks, I would would train the algorithm $\frac{18921}{4519}$ more vigorously on class "1" (as a learning rate multiplier), and would do a similar trick during test time. Basically Importance Sampling. Here is a good video about it: youtube.com/watch?v=ymeSc0SPFmE $\endgroup$ – Kari Nov 29 '18 at 12:52
  • 1
    $\begingroup$ "I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class." This is absolutely false. Models that predict probabilities make no such assumptions. See here: stats.stackexchange.com/questions/283170/… and stats.stackexchange.com/questions/312780/… $\endgroup$ – Matthew Drury Nov 29 '18 at 18:36
  • $\begingroup$ Thanks for pointing at these two discussions, I didn't know then and there are useful insights. However, data imbalance is indeed a problem for machine learning algorithms, please see: link.springer.com/content/pdf/10.1007%2Fs13748-016-0094-0.pdf. The effect could be less on probabilistic algorithms (not gonna argue that), but Random Forest is an ensemble learning algorithm, so it has no built-in capacity for probability or to deal with imbalanced distributions (and returning the probability of belonging to a class, is not really working with "probabilties"...). Thanks anyway. $\endgroup$ – Irene Dec 12 '18 at 8:46
2
$\begingroup$

The answer to your first question:

should the test dataset be balanced as well?

is, like many answers in data science, "it depends."

And really, it depends on the audience for, and interpretability of the model metrics, which is the thrust of your second question:

What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Personally, if the metrics will just be used by you to evaluate the model, I would use the sensitivity and specificity within each class to evaluate the model, in which case, I care less about the balance of the classes in the test data as long as I have enough of both to be representative. I can account for the prior probabilities of the classes to evaluate the performance of the model.

On the other hand, if the metrics will be used to describe predictive power to a non-technical audience, say upper management, I would want to be able to discuss the overall accuracy, for which, I would want a reasonably balanced test set.

That said, it sounds like your test set is drawn independently of the training data. If you are going to balance the training data set, why not draw one balanced data set from the raw data and then split the training and test data? This will give you very similar class populations in both data sets without necessarily having to do any extra work.

$\endgroup$
1
$\begingroup$

There are several ways of balancing classes. Either you can increase the number of samples of minority class or decrease the number of samples of majority class. Once you have balanced your classes, for a start you take 80% of it as train and test it on remaining 20%. In your case 4519 samples are 1 and 18921 are 0 so lets say you upsample 4519 and now you have 15,000 samples of class 1. You will take 80% of (15000+18921) and train your model. Test it on remaining 20%.

Recall becomes important when you are let's say identifying fraudulent transactions out of several millions. You will have very few classes labelled as 1 (fraudulent). So even you simply label all the classes in your test data as 0 you will be more than 99% accurate but you should ask yourself the question "Out of all the actual classes with label 1 how many did i get right from the trained model". That is what recall gives you.

$\endgroup$
1
$\begingroup$

Your model could have very dissimilar performances at detecting "0" or "1" samples without you noticing if the performance is not high enough.

I would include more samples in the training and subsample "0" (to keep balance of "0" and "1" in each step) during the training and would evaluate the model with a balanced test set.

$\endgroup$
1
$\begingroup$

A fast solution: If you use SKLearn's Random Forest, I highly suggest "class_weight" => “balanced”.

The argument will automatically weigh classes inversely proportional to their frequency.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.