I am implementing the CNN classification model. I've seen several instances where a dataset is split into test, train, and validation. I've also seen instances where a dataset is split into train and test, with the test data being used again for validation. What is the best approach to take here?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You generally need all 3. How you implement that will depend on your goals and how much compute / time you have. For a large CNN on a large dataset, it would be reasonable to hold back one test set, one validation set, and one larger training set. You need the val set to validate the CNN model while training. You will likely adjust hyperparameters within the CNN such as number of layers, strides, kernels, optimization algorithms, batch sizes, etc.. and you will use your validation set to validate your adjustments on 'unseen' data. You will probably also use it to know when to stop training your model so it doesn't overfit (early-stopping).
In the end though, all of that tuning means that your validation set is not purely unseen. You tuned your model using it in one way or another. So, you also need a completely naive test set to assess the final performance of your classifier model.
An alternative is to do some kind of nested-cross validation where you create repeated test sets and for each of those repeated validation and training sets. In that way you can get some bounds on the performance of your model (average and std. deviation of the accuracy). Of course, this means training your model over and over again and can get quite computationally expensive in deep-learning applications, so you will often just see 1 held out set for each of the 3.
