1
$\begingroup$

I have a Pytorch regression model as follows:

    model = nn.Sequential(nn.Linear(n_in, n_h_1),
                      nn.ReLU(),
                      nn.Dropout(p=0.1),
                      nn.Linear(n_h_1, n_h_2),
                      nn.ReLU(),
                      nn.Linear(n_h_2, n_out))

As you can see, I have used a Dropout regularization layer with dropout probability of 0.1. When the training is completed, we should disable the dropout. This can be achieved using model.eval(). Reference to this StackOverflow answer and other resources, we should multiply the output of hidden layer with (1-p) during inferencing of model.

My question is: do we have to do this manually using pytorch or the library handles this itself after applying model.eval()? If we have to do it ourselves, how can we achieve thisfor the given model?

$\endgroup$
1
$\begingroup$

PyTorch handles this with scaling the output of the dropout layer at training time with this probability:

enter image description here

So it handles this itself after applying model.eval(). You can lookup their code here:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.