23
$\begingroup$

I am new to pytorch and started with this github code. I do not understand the comment in line 60-61 in the code "because weights have requires_grad=True, but we don't need to track this in autograd". I understood that we mention requires_grad=True to the variables which we need to calculate the gradients for using autograd but what does it mean to be "tracked by autograd" ?

$\endgroup$
26
$\begingroup$

The wrapper "with torch.no_grad()" temporarily set all the requires_grad flag to false. An example from the official PyTorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) :

x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)

with torch.no_grad():
    print((x ** 2).requires_grad)

Out:

True
True
False

I recommend you to read all the tutorials from the website above.

In your example : I guess the author don't want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.

$\endgroup$
6
$\begingroup$
with torch.no_grad()

will make all the operations in the block have no gradients.

In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally change w1 and w2.

However, if you use this no_grad(), you can control the new w1 and new w2 have no gradients since they are generated by operations, which means you only change the value of w1 and w2, not gradient part, they still have previous defined variable gradient information and back propagation can continue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.