0
$\begingroup$

Here I overcame a problem about backward, below is a simple example written by python code.

enter image description here

the print information is that

enter image description here

This really confused me as self.weights is used as a important part in my transformer_based project, while i meets this problem, i try puts self.weights into torch.nn.paramaters but failed, could somebody give me a hint? Thanks!

$\endgroup$

1 Answer 1

0
$\begingroup$

When you index into a tensor, you are actually creating a new data object. The new data object has a computational graph linking it to the weights tensor via the select operation, but they are not the same thing.

weights = torch.ones(3, requires_grad=True)
print(weights)
> tensor([1., 1., 1.], requires_grad=True)

print(weights[0])
> tensor(1., grad_fn=<SelectBackward0>)

print(weights.data_ptr())
> 94109293078336

print(weights[0].data_ptr())
> 94109293078336

When you try to access weights[i].grad, you should get the error:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward().

This is because you are trying to access the grad attribute of a different object.

To access the gradient, you need to use the leaf variable:

weights = torch.ones(3, requires_grad=True)
a = 2 * weights[0]
a.backward()
print(weights.grad)
> tensor([2., 0., 0.])

If you want to access a slice of the gradient, you need to slice the gradient of the leaf tensor:

weights = torch.ones(3, requires_grad=True)
a = 2 * weights[0]
a.backward()
print(weights.grad[0])
> tensor(2.)
$\endgroup$
1
  • $\begingroup$ Thanks Karl! Now i get the correct understanding of leaf variable in the computational graph $\endgroup$
    – Ecthelion
    Mar 30 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.