I have:
training dataset of size 150k.
validation dataset of size 19k.
At each epoch I randomly sample without replacement 10k datapoints for training because I get Out of Mem Errors.
I need to downsample my validation set too. Which of the following methods seem most appropriate:
Two main things here:
Having a validation set of 19K samples is quite large. It's not bad, but it also means that unless you're dealing with a very complex problem, a reasonable sample out of the 19K should preserve the overall distribution of the 19K.
The way I would handle it is as follows. If you sample 10% of your training set, you should sample 10% of your validation set. That way, the proportions remain the same. You could split both datasets into 10 "groups" and iterate through each "train/validation" pair. With this amount of data, I don't expect huge issues when it comes to how representative your sets would be.
If you need to sample 10K items out of 150K to avoid out-of-memory errors, it might mean that you'd need to lazy load your data as you train, so that sampling isn't necessary anymore. If you don't run out of memory, but it just takes too long to run the validation, I'd indeed sample a small batch. Your validation set estimates the unknown distribution of your problem; whether you have 1K, 2K, or 19K items in that sample, it's still an estimate.
Actually you should never use any sampling techniques on your testing/evaluation data because this could lead to wrong classification results. If your dataset is imbalanced you could perform upsampling or downsampling techniques (like SMOTE) on your training data only. If you want to benchmark your multi-class classification you need to rely on e.g. the confusion matrix, recall, precision and F1 measure. Please keep in mind that the accuracy measure cannot be interpreted if you have too imbalanced data.
