I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.
2
-
$\begingroup$ Does your model progress in a loop? similar to neural networks? In that case you have a different model after each iteration and validation set can be used to keep the best model (at a specific iteration). Otherwise, you have only one model and validation set has no use. $\endgroup$– EsmailianMar 3, 2019 at 17:15
-
$\begingroup$ What do you mean to state by 'algorithm does not have any hyperparameter?'. Can you please elaborate on your problem. $\endgroup$– thanatozApr 3, 2019 at 10:34
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.
But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.
$\begingroup$
$\endgroup$
Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms. One will eventually perform well on the test set just due to the nature of that particular data. You need the final holdout set to get a less biased estimate.
Comparing models can be looked at the same way as tuning hyperparameters. Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.
