I have a dataset which is relatively small (less than a 1000 samples). I run an autoencoder on a training set, and then check the reconstruction error on a validation set and stop training before it increases. I then want to use the encoder to produce cluster-able representations of my high-dimensional input data.
But can I reliably encode all of my data now, or only my validation set – or should I keep an entirely separate test set, on which I can only expect reliable clustering on? In the case that training and validation errors are very similar, would there be anything wrong?
For the sake of the argument, let's say that I won't get anymore data, meaning that I can't just train my model, then wait for tomorrow for some new unseen data, on which I can apply my already-trained encoder on.