Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same weights with different hyperparameters), and Test set is used for evaluating our model". If true, can someone explain the whole process in a little more detail. TIA.


1 Answer 1


Not completely true. In validation set, we find the best hyperparameters, but not with the same parameters of the model. That is, for every value of the hyperparameters we run the training process and obtain the loss for that set of hyperparameters, and we select the hyperparameters values with the lowest validation loss.


Ridge regression example: Ridge regression depends on one parameter, $\lambda$, that penalizes your model weights. Ridge regression has a single hyperparameter. As ridge regression is a linear model, it has also some weights $\theta_0, \dots, \theta_n$ (parameters) to train. The way to proceed to choose $\lambda$ and $\theta_0, \dots, \theta_n$ is the following:

  • For each value of $\lambda$ in a grid, we train by gradient descent the ridge regression model (thus optimizing the training loss) and obtain a set of parameters $\theta_0^*, \dots, \theta_n^*$. We measure the loss of the trained ridge model in the validation set.
  • We choose the $\lambda$ that minimizmes the loss in the validation set and the parameters $\theta_0^*, \dots, \theta_n^*$. If we wish, and we don't have much data, we can retrain the ridge model with the optimal $\lambda$ in the concatenation of the training and validation sets, obtaining some parameters $\hat\theta_0, \dots, \hat\theta_n$.
  • We compute the test error with the parameters obtained in the last step and report that as the most realistic value of the error of the model.
  • $\begingroup$ So, what you are saying is, in validation step, we fine tune our hyperparameters, but the parameters we obtained in the training step also tend to change. Have I understood this correctly? $\endgroup$ Jun 7, 2018 at 7:58
  • $\begingroup$ Yes, they have to change! $\endgroup$ Jun 7, 2018 at 7:59
  • $\begingroup$ Yes, but I'm a little confused. If we update our weights (parameters) and hyperparameters in validation step, then why can't we use whole of training data with lots of hyperparameters in the beginning itself. The one which gives the least value of the loss function is the required set of parameters. What makes the validation step different? $\endgroup$ Jun 7, 2018 at 8:28
  • $\begingroup$ I have edited the answer $\endgroup$ Jun 7, 2018 at 8:36
  • $\begingroup$ Is it more clear? $\endgroup$ Jun 7, 2018 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.