Training model on a Balanced vs Imbalanced dataset?

Question

Let's say that I have a 2-class classification problem where classes A & B have 10*N and N observations respectively.

I am pretty sure that the answer to my question depends on the specific classification problem and on the features of my dataset etc.. Still there are general analysis that can be done on my question.?

Something that I could clarify is that I am interested in having high recall in both classes ("macro-average recall"); not primarily in having the highest possible recall in the minority class as in imbalanced dataset classification problems such as spam detection, financial fraud detection or disease detection.

So for this, generally speaking, is it better to train my model on:

1) A: 10*N observations, B: N observations

2) A: 5*N observations, B: N observations

3) A: N observations, B: N observations

I am having an impression, that assuming we start with a balanced dataset then the more data you add to one class then the better the macro-average recall because of the new information added but after one point the dataset becomes so imbalanced that the performance of the model on the minority class probably deteriorates and hence the macro-average recall falls.

Am I right on this?

Answer 1

It's an interesting problem, but I think the answer might be disappointing:

Usually problems about maximizing recall are considered in the context of a trade-off with precision, i.e. the goal is to sacrifice some precision by predicting more positives, whether true (TP) or false (FP). Usually in a binary problem this is possible because we focus on one class of importance and consider the other as irrelevant, so its performance doesn't matter.

Here we have a binary problem where we want to maximize recall for both classes, so we cannot sacrifice one for the sake of the other: any gain in recall for a class is likely to cause a loss in recall for the other class, since predicting more instances as positive for class A would mean more negative instances for class B, and conversely. If we were talking micro-recall we could still use the higher importance in proportion of class A, but with macro-recall we can't. Note that this is a typical case where accuracy could be used, since it would give the same weight to both classes and would be a much simpler metric.

So the only way to improve macro-recall is to increase true positives. Let's look at the options:

1) A: 10*N observations, B: N observations

Pro: performance for class A is maximized. Cons: class B proportionally disadvantaged, so possible loss in its performance.

2) A: 5*N observations, B: N observations

Cons: class A a bit less advantaged; pro: class B a bit less disadvantaged.

3) A: N observations, B: N observations

Pro: maximum performance for class B; cons: class A underperforms.

Actually the best option is probably to use all of N*10 instances for A and repeat 10 times the N instances for B, so that:

• the learning method can benefit from all the available training data for A
• class B is not proportionally disadvantaged.

But my guess is that it's unlikely to make a very big difference anyway. When it's a matter of increasing true positives, it's usually the features and the ML method which can have an impact.

Answer 2

You can also try CalibratedClassifierCV if your data is imbalanced. The plots have been really useful to get the insights of the data.