# How do we implement a custom loss that backpropagates with PyTorch?

In a neural network code written in PyTorch, we have defined and used this custom loss, that should replicate the behavior of the Cross Entropy loss:

def my_loss(output, target):
global classes

v = torch.empty(batchSize)
xi = torch.empty(batchSize)

for j in range(0, batchSize):
v[j] = 0
for k in range(0, len(classes)):
v[j] += math.exp(output[j][k])

for j in range(0, batchSize):
xi[j] = -math.log( math.exp( output[j][target[j]] ) / v[j] )

loss = torch.mean(xi)
print(loss)
return loss


but it doesn't converge to accetable accuracies.

You should only use pytorch's implementation of math functions, otherwise, torch does not know how to differentiate them. Replace math.exp with torch.exp, math.log with torch.log.

Also, try to use vectorised operations instead of loops as often as you can, because this will be much faster.

Finally, as far as I can see, you are merely reimplementing a log loss in pytorch, any reason why you don't use one that has been implemented by default? (see here or here)

[EDIT]: If after having removed math operations and implemented a vectorised version of the loss, it still does not converge, here are a few pointers on how to debug it:

• Check that the loss is correct by calculating the value manually and compare it with what the function outputs
• Compute the gradient manually and check that it is the same as the values in loss.grad, after running loss.backward() (more info here)
• Monitor the loss and the gradient after a few iterations to check that everything goes right during the training
• I've changed all the math. functions to torch. calls. Still it does not converge. Jun 3 '20 at 7:43
• Have you checked that your model converges with the Crossentropy loss provided by pytorch? If not, then the model or the data is in cause, not the loss Jun 3 '20 at 7:45
• yes, it does converge with the Crossentropy loss provided by pytorch. Jun 3 '20 at 7:46
• Then reimplement with a vectorised version of your loss and drop the loops Jun 3 '20 at 7:48
• Sorry, but I'm ignorant on how to do it with Python and PyTorch. Jun 3 '20 at 7:50