Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same weights with different hyperparameters), and Test set is used for evaluating our model". If true, can someone explain the whole process in a little more detail. TIA.
Not completely true. In validation set, we find the best hyperparameters, but not with the same parameters of the model. That is, for every value of the hyperparameters we run the training process and obtain the loss for that set of hyperparameters, and we select the hyperparameters values with the lowest validation loss.
Ridge regression example: Ridge regression depends on one parameter, $\lambda$, that penalizes your model weights. Ridge regression has a single hyperparameter. As ridge regression is a linear model, it has also some weights $\theta_0, \dots, \theta_n$ (parameters) to train. The way to proceed to choose $\lambda$ and $\theta_0, \dots, \theta_n$ is the following:
- For each value of $\lambda$ in a grid, we train by gradient descent the ridge regression model (thus optimizing the training loss) and obtain a set of parameters $\theta_0^*, \dots, \theta_n^*$. We measure the loss of the trained ridge model in the validation set.
- We choose the $\lambda$ that minimizmes the loss in the validation set and the parameters $\theta_0^*, \dots, \theta_n^*$. If we wish, and we don't have much data, we can retrain the ridge model with the optimal $\lambda$ in the concatenation of the training and validation sets, obtaining some parameters $\hat\theta_0, \dots, \hat\theta_n$.
- We compute the test error with the parameters obtained in the last step and report that as the most realistic value of the error of the model.