Forcing class imbalance to mirror the target data

Question

I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B.

Is it a worthwhile strategy to use SMOTE to over-sample A such that I'm training on a class split that mirrors the data I'm trying to classify? The only metric that concerns me is accuracy.

Answer 1

Disclaimer: I am assuming that by "source data" you mean the data you are using for training and by "target data" you mean the real world data the model will aim to classify.

SMOTE Is usefull when your data is highly imbalance i.e. 1% A/ 99% B or something around this proportions. In your case I would expect the model to work just fine when classifying a different distribution of data as long as you have enough training examples in each class. If this approach doesn't work you can use cost sensitive learning to fit the imbalance but I would consider this a secondary path

Answer 2

The imbalance in your training and target data is not too large; I agree with Ismor that SMOTE will probably do more harm than good in this case.

The best approach will probably depend on how much data you have:

• Subsample the training data by stratified random sampling along the value of the target to match the distribution of the target in your target data. In your case, that means losing well over one half of the B-labeled training samples.
• Calibrate the output probabilities to the class distribution that appears in your target data. This means using the whole set to train the main classifier and then fitting another classifier (usually a logistic regressor since it it provides a simple transformation) on the subsampled dataset as described above, using just the raw positive class probability from the main classifier as a single feature. This should give you a distribution of the probability that matches the class proportions of your target data.