5
$\begingroup$

When using machine learning models like gradient boosted trees and CNN, is it required (or considered as an always-do good practice) to balance the amount of positive/negative examples when learning for binary classification?

Given P positive examples and N negative examples, where P << N, I can think of several choices: (Let's forget about validation set and test set)

Choice A) No balancing at all, put all examples (totally P+N) into the training set without weighting w.r.t. their ratio.

Choice B) Put all examples (totally P+N) into the training set, but weight all positive examples 1/2P and all negative examples 1/2N, so that total weight of positive examples and negative example equal.

Choice C) Take all P positive examples, then sample P negative examples (out of N), and train with these 2P examples with uniform weighting.

What are the pros/cons for each of the approach and which one(s) do we usually go with?

$\endgroup$
6
  • 1
    $\begingroup$ Are the two classes equally important? If not, how important is one class relative to the other and what fraction of the sample does it constitute? This is what I want to know before I start. $\endgroup$
    – Emre
    Apr 22 '17 at 1:33
  • $\begingroup$ Hi @Emre, thank you for your interest in answering. What do you mean by "importance of the classes"? Do you mean tolerance to TP/FP for each class? In other words, if the model has to make a mistake, which kind of mistake I prefer? I'd imagine this will be useful when looking at a P-R curve and choosing an operating point, but could you help me understand why it help in training? $\endgroup$
    – Roy
    Apr 25 '17 at 18:01
  • 1
    $\begingroup$ Because that will determine the loss function. In any case you should train with all the data. $\endgroup$
    – Emre
    Apr 25 '17 at 18:24
  • $\begingroup$ The two classes are about 1:10 in their number of examples. Each example is associated w/ a floating-point-valued reward (or penalty when negative). My goal is to maximize the sum of regard/penalty of triggered examples. Score cutoff tuning is allowed if necessary. Does this answer your question? $\endgroup$
    – Roy
    Apr 25 '17 at 19:58
  • $\begingroup$ More clarification: The overall sum of all rewards and all penalties are mostly the same (totally sum up to approximately zero if we add up all examples). And the number of rewarded and penalized examples are mostly the same, too. The reason that I labeled them 1:10 is that I wanted the classifier to be safer (prefer FN over FP), so I only assigned positive label to the highly rewarded examples. This means, quite some (weakly) rewarded examples are assigned to the negative class. I realize this might be a mistake, i.e. maybe I should have labeled all rewarded examples positive and vice versa? $\endgroup$
    – Roy
    Apr 25 '17 at 20:04
3
$\begingroup$

Let's start by answering your first question. Is it required to balance the dataset?

Absolutely, the reason is simple in failing to do so you end up with algorithmic bias. This means that if you train your classifier without balancing the classifier has a high chance of favoring one of the classes with the most examples. This is especially the case with boosted trees. Even normal decision trees, in general, have the same effect. So it is always important to balance the dataset

Now let's discuss the three different scenarios placed.

Choice A): This would be what I explained all along. I'm not saying necessarily you will have a bias. It depends on the dataset itself. If the nature of the dataset has a very fine distinction with the boundaries then the chance of misclassification is reduced, you might get a decent result but it's still not recommended. Also if the data does not have good boundaries then the rate of misclassification rises a lot.

Choice B): Since you are placing weights for each sample you are trying to overcome the bias with a penalty. This is also called as an Asymmetric method. Normally these methods increase the accuracy of a model by a slight margin but that mostly depends on the machine learning algorithm you are using. In examples like Adaboost such a model the effectivity of the model increases. This method is also called Asymmetric Adaboost. But this might not necessarily work with all algorithms.

Choice C): Assuming you have weighted the samples accordingly it should do the same as either choice A or choice B. I'll leave this for you to extrapolate based on my previous explanations.

$\endgroup$
3
  • $\begingroup$ Thank you so much for the detailed explanation, @rahul-aedula. I wanted to ask a follow-up question if I may: Does the choice of the training objective matter here? If we use AUC of ROC (e.g. as in xgbclassifier.fit(..., eval_metric='auc', ...)), since AUC is insensitive to imbalanced classes, does that mean the 'algorithmic bias' problem will not occur? $\endgroup$
    – Roy
    Apr 25 '17 at 3:40
  • $\begingroup$ It's not that the algorithmic bias will not occur, it is more of a case that it shows better accuracy despite imbalanced classes. It is always better to have balanced classes nevertheless. $\endgroup$ Apr 25 '17 at 10:47
  • $\begingroup$ Side note: AUC - ROC has come into question for not being a good reliable method to determine the efficiency of a binary classifier. It has a few mathematical problems. I urge you to go read about it in detail. $\endgroup$ Apr 25 '17 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.