Let's say that I have a 2-class classification problem where classes
A & B have
N observations respectively.
I am pretty sure that the answer to my question depends on the specific classification problem and on the features of my dataset etc.. Still there are general analysis that can be done on my question.?
Something that I could clarify is that I am interested in having high recall in both classes ("macro-average recall"); not primarily in having the highest possible recall in the minority class as in imbalanced dataset classification problems such as spam detection, financial fraud detection or disease detection.
So for this, generally speaking, is it better to train my model on:
1) A: 10*N observations, B: N observations
2) A: 5*N observations, B: N observations
3) A: N observations, B: N observations
I am having an impression, that assuming we start with a balanced dataset then the more data you add to one class then the better the
macro-average recall because of the new information added but after one point the dataset becomes so imbalanced that the performance of the model on the minority class probably deteriorates and hence the
macro-average recall falls.
Am I right on this?