2
$\begingroup$

I have a multi-class classification problem to solve which is highly imbalanced. Obviously I'm doing oversampling, but I'm doing cross-validation with the over-sampled dataset, as a result of which I should be having repetition of data in the train as well as validation set. I'm using lightgbm algorithm, but surprisingly there is not much difference between cross-validation score and the score on the unseen dataset.

However I just want to know whether its fine to do cross-validation after oversampling the dataset, if not why am I getting such close score on the validation set and the unseen test set?

Also if its not correct to do oversampling before the cross-validation, then it becomes to lengthy to split the data into validation and training and then again sample the training set, and again during final prediction if you're looking to use all the data then you've to append the validation and the training data and then again oversample. Is there any shortcut method to solve the problem?

$\endgroup$
2
$\begingroup$

Oversampling the training data may help the classifier to better predict on the originally less represented class. This does not mean that it should be applied to performance metrics, as it changes the original target distribution and thus creates bias in the results.

Imagine the problem of cancer detection, where your original dataset is unbalanced: 10% of the patients have cancer y=1 and the remaining 90% don't y=0. If you train a classifier which is prone to error on unbalanced datasets (such as an Artificial Neural Network), you may end up predicting always the majority class: y=0.

If you oversample to a new distribution, let's say 50/50, your classifier is expected to increase the performance, specially on the positive class. Nonetheless, to measure the performance on real data, which is by itself skewed, measure on oversampled may not be the best choice.

Thus, if you are optimizing the hyperparameters or choosing from a set of classifiers, cross-validating with oversampled data may provide you with a different perspective on the classifier's ability to predict on both classes with equal importance. Nonetheless, if you are estimating the real-life prediction capability, I would not advise you to oversample such validation data!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.