My dataset has 3 class and 900 examples for training. Class distribution is 255, 185, and 460.

I found that if I oversample (random) the training data then I have to correct/calibrate the predicted probability of the test data because after oversampling the training and testing data distribution are not same. This is nicely described here, here and here

I have 4 questions:

Should we do this to calculate:

Training loss

My guess: We should not change the posterior/predicted probability for training loss calculation. Because that will cancel out the effect that has been created by oversampling. Am I right?

validation loss My guess: Since the validation loss has no effect on training (optimization), we can correct the predicted probabilities but then it would not be comparable with the training loss. Since we compare both losses to check overfitting, we should not do any probability correction here. Unless we compute two kind of training losses. One is without probability correction which would be used to compute the gradient and another corrected training loss which would be used to plot with corrected validation loss. Then these two losses would be comparable. Am I right?

validation accuracy

My guess: For validation accuracy, I think we should do it for the same reason we have to do it on testing data. Right?

Is this a mandatory step? I am asking this because this might hurt the overall accuracy. Because this will penalize the probabilities of the classes which have less example.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.