1
$\begingroup$

Fairly newbie to Pytorch & neural nets world.Below is a code snippet from a binary classification being done using a simple 3 layer network :

n_input_dim = X_train.shape[1]
n_hidden = 100  # Number of hidden nodes
n_output = 1   # Number of output nodes = for binary classifier
# Build the network
model = nn.Sequential(
    nn.Linear(n_input_dim, n_hidden),
    nn.ELU(),
    nn.Linear(n_hidden, n_output),
    nn.Sigmoid())

x_tensor =  torch.from_numpy(X_train.values).float()
tensor([[ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ...,   0.1538,   5.0000,   0.1538],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000,   6.0000,   0.2381],
       ...,
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000],
       [ -1.0000,  -1.0000,  -1.0000,  ..., -99.0000, -99.0000, -99.0000]])
y_tensor =  torch.from_numpy(Y_train).float()
tensor([0., 0., 1.,  ..., 0., 0., 0.])
#Loss Computation
loss_func = nn.BCELoss()
#Optimizer
learning_rate = 0.0001
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

train_loss = []
iters = 500
for i in range(iters):
    y_pred = model(x_tensor)
    loss = loss_func(y_pred, y_tensor)
    print " Loss in iteration :"
    print (i, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    train_loss.append(loss.item())

In the above case , what i'm not sure about is loss is being computed on y_pred which is a set of probabilities ,computed from the model on the training data with y_tensor (which is binary 0/1). Is this way of loss computation fine in Classification problem in pytorch? Shouldn't loss be computed between two probabilities set ideally ? If this is fine , then does loss function , BCELoss over here , scales the input in some manner ?

Any insights towards this will be highly appreciated

$\endgroup$
  • $\begingroup$ The optimizer.zero_grads() is on a completely wrong point. It should be right after the for. $\endgroup$ – Dimitris Lolis Dec 17 '19 at 14:49
  • $\begingroup$ @DimitrisLolis Can you elaborate on that? Should't the grad call typically be before every step call? $\endgroup$ – basket Jan 14 at 22:25
1
$\begingroup$

You are right about the fact that cross entropy is computed between 2 distributions, however, in the case of the y_tensor values, we know for sure which class the example should actually belong to which is the ground truth. So, you can think of the binary values as probability distributions over possible classes in which case the loss function is absolutely correct and the way to go for the problem. Hope that helps.

$\endgroup$
0
$\begingroup$

Adding to the answer given by @Sajid Ahmed, you can also see from the PyTorch documentation that it, indeed, works just fine. The example they provide is:

import torch
import torch.nn as nn
m = nn.Sigmoid()
loss = nn.BCELoss()
input = torch.randn(3, requires_grad=True)
target = torch.empty(3).random_(2)
output = loss(m(input), target)
output.backward()

For which

print("input: \t\t{}".format(input))
print("m(input): \t{}".format(m(input)))
print("target: \t{}".format(target))
print("output: \t{}".format(output))

gives the following output:

input:      tensor([ 1.3313,  1.2612,  0.3466])
m(input):   tensor([ 0.7911,  0.7792,  0.5858])
target:     tensor([ 0.,  1.,  1.])
output:     0.7833071351051331

As you can see it works as desired.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.