Compare model accuracy when training with imbalanced and balanced data

Question

So I was recently doing a data science project which is a multi class classification. The project can be found https://www.kaggle.com/c/otto-group-product-classification-challenge.

The dataset is an imbalanced dataset with 93 features and 9 possible outcomes (targets).

Since we don't know what any of these features are we don't know what kind of categories the targets represent I am not sure if balancing the data before training the model makes sense.

Therefore I just trained each of my test models with both, once with a balanced and once with an imbalanced dataset.

In particular this is what I did:

1. do a simple 80/20 split for training and test to create an imbalanced data and training set

index <- createDataPartition(data$target, p=.8, list=FALSE,times=1)
training <- data[index,]
test <- data[-index,]

2. downsample the training split and use it to create a downsampled training set and use the rest of the data for testing

training.downsampled <- downSample(training[,-ncol(training)],y=training$target,yname="target")
test.downsampled <-subset(data, !(id %in% training.downsampled$id))

So now to come to my main question. If I now train a model, for example a random forest, can I use the accuracies of both to compare if the model delivers a better accuracy while using balanced data? I am concerned since I test against more data for the balanced one. If I can't compare it like this, then what would be a suitable method to achieve comparison of the both.

Answer 1

Accuracy is the worst metric you could use for an imbalanced dataset. If you choose accuracy as a metric when you have class imbalance, you will get very high accuracy. This is because the majority class has a higher frequency (or has more number of records) and hence the model will predict the majority class as the prediction majority of the time.

The metric you choose depends on what kind of dataset you have. If your data has class imbalance, you can go for F1 score, AUC score, True positive/True negative rate. They will give a more realistic score rather than accuracy.

Another point to remember is that if you want to balance your dataset, never use downsampling as it results in data loss which is a BIG NO NO. Always use oversampling.

A word of caution though. Some experts believe that undersampling or oversampling is not the way to go when dealing with imbalance. Rather choosing the right metric is enough to deal with it. But other experts say that SMOTE is the way to go. It depends on you on what you think is right although comparing models like you are doing is probably a safe bet.

Other than that you are correct in your procedure to compare both the models.