2
$\begingroup$

I am trying to understand how Binary Cross Entropy is counted in PyTorch. I've tried the same code from the PyTorch documentation here, but I get a different result comparing to mathematical implementation of this function.

Code (I made little changes comparing to documentation):

import torch
import torch.nn as nn
m = nn.Sigmoid()
loss = nn.BCELoss()
input = torch.randn(1, requires_grad=True)
print(input)
print(m(input))
target = torch.empty(1).random_(2)
print(target)
output = loss(m(input), target)
print(output)
output.backward()
print(output)

Output for the code above is: enter image description here

The mathematical formula for BCELoss is:

output = -[target*log(our_value) + (1-target)*log(1-our_value)]

From the mathematical formula above I should get 'output'=0.3215 for 'our_value'=0.4770 and 'target'=1. But PyTorch shows that the 'output'=0.7403.

I've also found a C code here of this function but the formula is almost the same (only very small epsila should be added and it gives no difference in output). So, it's still unclear for me.

How does it come to? Any hints are welcome.

$\endgroup$
1
$\begingroup$

Take a look at the documentation for BCELoss

Our solution is that BCELoss clamps its log function outputs to be greater than or equal to -100. This way, we can always have a finite loss value and a linear backward method.

If you amend the definition to include that it should yield the correct results:

def binary_cross_entropy(pred, y): 
    log_pred = torch.clamp(torch.log(pred), -100, 100)
    log_pred_rev = torch.clamp(torch.log(1-pred), -100, 100)
    return -(log_pred*y + (1-y)*log_pred_rev).sum()


a = torch.linspace(0, 1, 10)

w = np.array([F.binary_cross_entropy(x,torch.tensor(0.2)).item() for x in a])
q = np.array([binary_cross_entropy(x,torch.tensor(0.2)).item() for x in a])

print(w)
print(q)

Gives

[20.          0.53367138  0.501867    0.54409462  0.63241535  0.76630157 0.95998287  1.2535249   1.78133631 80.        ]
[20.          0.53367138  0.501867    0.54409462  0.63241535  0.76630157 0.95998287  1.2535249   1.78133631 80.        ]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.