So I was recently doing a data science project which is a multi class classification. The project can be found https://www.kaggle.com/c/otto-group-product-classification-challenge.
The dataset is an imbalanced dataset with 93 features and 9 possible outcomes (targets).
Since we don't know what any of these features are we don't know what kind of categories the targets represent I am not sure if balancing the data before training the model makes sense.
Therefore I just trained each of my test models with both, once with a balanced and once with an imbalanced dataset.
In particular this is what I did:
- do a simple 80/20 split for training and test to create an imbalanced data and training set
index <- createDataPartition(data$target, p=.8, list=FALSE,times=1) training <- data[index,] test <- data[-index,]
- downsample the training split and use it to create a downsampled training set and use the rest of the data for testing
training.downsampled <- downSample(training[,-ncol(training)],y=training$target,yname="target") test.downsampled <-subset(data, !(id %in% training.downsampled$id))
So now to come to my main question. If I now train a model, for example a random forest, can I use the accuracies of both to compare if the model delivers a better accuracy while using balanced data? I am concerned since I test against more data for the balanced one. If I can't compare it like this, then what would be a suitable method to achieve comparison of the both.