0
$\begingroup$

In PyTorch when we call loss.backward() it performs backpropagation for the sample (for stochastic case). Let’s consider my output is 50 dimensional. I have two loss components. First one is an array of dimension 50. How can I run loss.backward() for each dimension separately? I also have another loss component which is a scalar and I want to do normal backpropagation for this one on whole output tensor, together with my first loss on each dimension of the output. Do I have to use a custom loss function?

I have tried x.view() function to split weight matrix to each dimension for the last layer but I have not figured out how to do backpropagation for each dimension yet.

Thanks a lot for you kind help.

Nil

$\endgroup$

1 Answer 1

1
$\begingroup$

A typical training step looks like this in Pytorch:

for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

Change 1: Create a custom loss function [1]

Change 2: Compute and accumulate gradients. Gradients are zeroed out once, then gradients are calculated multiple times based on the number of outputs and finally optimizer.step() is called once.

for input, target in dataset:
    optimizer.zero_grad()
    outputs = model(input)

    for output in outputs:
       loss = my_loss_fn(output, target)
       loss.backward()
    
    optimizer.step()

The exact implementation will depend on how you compute loss for the vector components.

This assumes that my_loss_fn() can be implemented using other Pytorch functions. If that is not the case, you will have to create new Pytorch operations by extending the Function class[2].

$\endgroup$
1
  • $\begingroup$ Thanks for your response. My loss is a 50 dimensional array. My input and output are also 50 dimensional arrays. So, I want to do backpropagation for each dimension. I tried your code block but I did not work for my case. $\endgroup$ Aug 4, 2023 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.