How does the validation set get used in the training phase?

Question

I am confused about how the validation set is used during the training phase (neural network like CNN)? In a platform like Matlab or python(Keras), I split my dataset into train set, validation set and test set. I knew that validation set is used to tune hyperparameters(like the number of neurons and learning rate), suppose SDG optimizer is used, how the tuning hyperparameters happened based on validation set? Does the validation set just give an indicator of how the neural network performs on unseen data(validation set) then based on that I manually set hyperparameters? or something automatically(the optimizer) tune hyperparameters?

Answer 1

The three way - train, validation(dev), test split helps in unbiased evaluation of the model on unseen data , that is the test set. The train and validation sets are used for training weights and hyperparameter tuning respectively. In Keras if you have set the 'validation_split' parameter in 'model.fit' then you can look at the performance of your model on the validation set and tune the hyperparameters manually in order to get a good validation set performance.Once you are satisfied with the performance of your model on the validation set, you finally evaluate your model on the test set. Hyperparameter tuning is done manually and not automatically by Keras.

Answer 2

The validation set is used just to give an indication about the network's performance. The hyperparameters are tuned based on the training data itself.

Based on the optimizer, the neural network calculates the loss for a training sample. Based on the computed loss, the weights of the network are updated. This is repeated for every sample in the training set.

The validation set just uses the latest weights of the network on unseen data to check the model's performance.