# Compare model accuracy when training with imbalanced and balanced data

So I was recently doing a data science project which is a multi class classification. The project can be found https://www.kaggle.com/c/otto-group-product-classification-challenge.

The dataset is an imbalanced dataset with 93 features and 9 possible outcomes (targets).

Since we don't know what any of these features are we don't know what kind of categories the targets represent I am not sure if balancing the data before training the model makes sense.

Therefore I just trained each of my test models with both, once with a balanced and once with an imbalanced dataset.

In particular this is what I did:

1. do a simple 80/20 split for training and test to create an imbalanced data and training set
index <- createDataPartition(data$target, p=.8, list=FALSE,times=1) training <- data[index,] test <- data[-index,]  1. downsample the training split and use it to create a downsampled training set and use the rest of the data for testing training.downsampled <- downSample(training[,-ncol(training)],y=training$target,yname="target")
test.downsampled <-subset(data, !(id %in% training.downsampled\$id))


So now to come to my main question. If I now train a model, for example a random forest, can I use the accuracies of both to compare if the model delivers a better accuracy while using balanced data? I am concerned since I test against more data for the balanced one. If I can't compare it like this, then what would be a suitable method to achieve comparison of the both.

Accuracy is the worst metric you could use for an imbalanced dataset. If you choose accuracy as a metric when you have class imbalance, you will get very high accuracy. This is because the majority class has a higher frequency (or has more number of records) and hence the model will predict the majority class as the prediction majority of the time.

The metric you choose depends on what kind of dataset you have. If your data has class imbalance, you can go for F1 score, AUC score, True positive/True negative rate. They will give a more realistic score rather than accuracy.

Another point to remember is that if you want to balance your dataset, never use downsampling as it results in data loss which is a BIG NO NO. Always use oversampling.

A word of caution though. Some experts believe that undersampling or oversampling is not the way to go when dealing with imbalance. Rather choosing the right metric is enough to deal with it. But other experts say that SMOTE is the way to go. It depends on you on what you think is right although comparing models like you are doing is probably a safe bet.

Other than that you are correct in your procedure to compare both the models.

• Thanks a lot for you input. So using the F1_value for both the imbalanced training and balanced training should work? So I could make a statement using the f1 for example to tell which option (balanced or imbalanced) produces a better output? Oct 22, 2021 at 14:42
• Yes you can. Also don't use undersampling to balance your data. Oct 22, 2021 at 14:51
• Harrell would argue that threshold-based metrics are problematic whether there is class imbalance or not (1)(2), and Kolassa would argue that accuracy is problematic when the class balance is even (3). Why not use something like log loss that directly measures the probabilistic predictions and does not care about class imbalance?
– Dave
Oct 22, 2021 at 16:00
• "Always use oversampling", this is patently false. There are valid cases for downsampling, eg when data contain irrelevant items, or not sufficiently informative items, it is a good choice to downsample since information contained in the rest data is increased. Oct 22, 2021 at 17:09
• @NikosM. Usually we downsample or upsample when we have class imbalance and the final data where the irrelevant info has been removed. Downsampling then can lead to loss of info Oct 23, 2021 at 7:26