How is the validation set processed in PyTorch?

Question

Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the test set. The ML algorithm is a neural network.

The training set is processed (in minibatches) by the code below. First, one sets the gradients to zero. Then, the model makes a prediction, and the loss is calculated. Next, the gradients are computed, and the weights are updated via backpropagation.

def train(data, label):
    model.zero_grad()
    prediction = model(data)
    loss = loss_function(prediction, label)
    loss.backward()
    optimizer.step()
    return loss

As I understand, the validation set is used for hyperparameter tuning, whereas the test set is used for evaluation of the final model (as a reference to compare performance to other models). The accuracy on the test set is measured after "freezing" the model, like in the code below.

for parameter in model.parameters():
    parameter.requires_grad = False
model.eval()

So, my questions are:

• When processing the validation set, is it correct to use the code of train() or must one omit the backpropagation?

• If one assumes that a neural network applies dropout, is dropout enabled or disabled while processing the validation set?

Answer 1

With regards to the backpropagation, since the network is not trained on the validation set and parameters are not updated we do not have to use backpropagation. The calculation of gradients can be disabled by using the torch.no_grad() context manager from pytorch. By not calculation gradients the model is able to process data faster. Dropout is turned off when setting the model in evaluation mode using model.eval() and is often used in combination with torch.no_grad() to turn off the computation of gradients, see also this stackoverflow answer and the pytorch documentation.