35
$\begingroup$

I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?

$\endgroup$
33
$\begingroup$

The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :

x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)

with torch.no_grad():
    print((x ** 2).requires_grad)

Out:

True
True
False

I recommend you to read all the tutorials from the website above.

In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Thanks for this answer. However, I still have a doubt: why would you want to explicitly code that you do NOT want to track the variables with autograd? Is there a memory allocation reason or what? Thanks! $\endgroup$ – desmond13 Mar 30 at 8:36
  • $\begingroup$ Hey, I looked at the code and I can't see any problem with not having the torch.no_grad() line. I mean anyway we clear the grad so that tracking should not matter. Please correct me if I am wrong! $\endgroup$ – Black Jack 21 Apr 7 at 10:41
  • $\begingroup$ To add to this answer: I had this same question, and had assumed that using model.eval() would mean that I didn't need to also use torch.no_grad(). Turns out that both have different goals: model.eval() will ensure that layers like batchnorm or dropout will work in eval mode instead of training mode; whereas, torch.no_grad() is used for the reason specified above in the answer. Ideally, one should use both if in the evaluation phase. $\endgroup$ – Lakshay Sharma Apr 23 at 22:34
8
$\begingroup$
with torch.no_grad()

will make all the operations in the block have no gradients.

In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.

However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hey, I looked at the code and I can't see any problem with not having the torch.no_grad() line. I mean anyway we clear the grad so that tracking should not matter. Please correct me if I am wrong! $\endgroup$ – Black Jack 21 Apr 7 at 10:42
3
$\begingroup$

Torch.no_grad() deactivates autograd engine. Eventually it will reduce the memory usage and speed up computations.

Use of Torch.no_grad():

  • To perform inference without Gradient Calculation.

  • To make sure there's no leak test data into the model.

It's generally used to perform Validation. Reason in this case one can use validation batch of large size.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I think if we don't use torch.no_grad then weight update step will be added to the computation graph of the Neural Network which is not desired.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.