I have:
training dataset of size 150k.
validation dataset of size 19k.

At each epoch I randomly sample without replacement 10k datapoints for training because I get Out of Mem Errors.

I need to downsample my validation set too. Which of the following methods seem most appropriate:

  • Randomly sampling validation set which is x% of 10k and use the same set across every epoch.
  • Randomly sampling validation set which is x% of 10k at every epoch.
  • $\begingroup$ Do you really need to sample your validation set of 19k samples? Why so? Do you get OOM errors here, too? $\endgroup$ – desertnaut Nov 11 '20 at 1:24
  • $\begingroup$ It takes significant amount of time (2x the training time) per epoch if I use 19k samples. $\endgroup$ – Heisenbug Nov 11 '20 at 1:32

Actually you should never use any sampling techniques on your testing/evaluation data because this could lead to wrong classification results. If your dataset is imbalanced you could perform upsampling or downsampling techniques (like SMOTE) on your training data only. If you want to benchmark your multi-class classification you need to rely on e.g. the confusion matrix, recall, precision and F1 measure. Please keep in mind that the accuracy measure cannot be interpreted if you have too imbalanced data.

  • $\begingroup$ If you get out of memory errors you could use batch learning... this is usually done if you train on a huge dataset $\endgroup$ – Predicted Life Nov 11 '20 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.