0
$\begingroup$

I have:
training dataset of size 150k.
validation dataset of size 19k.

At each epoch I randomly sample without replacement 10k datapoints for training because I get Out of Mem Errors.

I need to downsample my validation set too. Which of the following methods seem most appropriate:

  • Randomly sampling validation set which is x% of 10k and use the same set across every epoch.
  • Randomly sampling validation set which is x% of 10k at every epoch.
$\endgroup$
2
  • $\begingroup$ Do you really need to sample your validation set of 19k samples? Why so? Do you get OOM errors here, too? $\endgroup$
    – desertnaut
    Nov 11, 2020 at 1:24
  • $\begingroup$ It takes significant amount of time (2x the training time) per epoch if I use 19k samples. $\endgroup$
    – Heisenbug
    Nov 11, 2020 at 1:32

1 Answer 1

0
$\begingroup$

Actually you should never use any sampling techniques on your testing/evaluation data because this could lead to wrong classification results. If your dataset is imbalanced you could perform upsampling or downsampling techniques (like SMOTE) on your training data only. If you want to benchmark your multi-class classification you need to rely on e.g. the confusion matrix, recall, precision and F1 measure. Please keep in mind that the accuracy measure cannot be interpreted if you have too imbalanced data.

$\endgroup$
2
  • $\begingroup$ If you get out of memory errors you could use batch learning... this is usually done if you train on a huge dataset $\endgroup$ Nov 11, 2020 at 11:31
  • $\begingroup$ You didn't say how your original sample was sampled in the first place. If it was simple random sampling, you can certainly take another random sample in the same ratio of training to test. But, based on your question, that would be the final sample. You didn't describe any goal in doing multi sampling. $\endgroup$ Apr 24 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.