Pytorch CrossEntropyLoss expected long but got float

Question

1. have you done some research before asking the question?

Yes. I have done a lot of online searching, and others had similar problems. There solution was to use .float() when entering into the loss function. This did not work for me. Instead, regardless if I even do .type(float.long) etc. I still get the same error. I predict it has something to do with the way that my Net is setup/outputting. But I honestly don't know for sure.

2. What have you done to try and solve the problem?

I have re-written my code, fact checked my methodology with a colleague, and also done some rubber-duck programming to no avail.

3. what language?

Python 3.7.5, PyTorch 1.3.1

4. short and sweet code?

I don't know how to properly share data, but long-story short the input has 66 features between [-1,1] (using PCA to decompose the MNIST image)

import torch
import torch.nn as nn
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(66, 99)
        self.output = nn.Linear(99, 10)

    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        x = x.sigmoid()
        return x


custom_dataset = CustomDataset()
train_loader = torch.utils.data.DataLoader(dataset=custom_dataset.MNIST_train,
                                           batch_size=64,
                                           shuffle=True)  # outputs (sample[], targets[]) -> (64x66, 64x1)
test_loader = torch.utils.data.DataLoader(dataset=custom_dataset.MNIST_test,
                                           batch_size=64,
                                           shuffle=True)  # outputs (sample[], targets[]) -> (64x66, 64x1)

n, target_size, num_epoc, learning_rate = 66, 1, 100, 0.001
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

for i in range(num_epoc):
    for x in train_loader:
        # run model and collect loss
        y = model.forward(x[0].float())
        loss = criterion(y, x[1].float())

        # perform optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(str(i) + ': ' + str(loss))

5. what error is thrown?

The program is throwing the following error:

 Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\pydevd.py", line 2060, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\pydevd.py", line 2054, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\pydevd.py", line 1405, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\pydevd.py", line 1412, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/User/Desktop/paper_recreation/PCA_CNN/CNN_DEBUG.py", line 109, in <module>
    loss = criterion(y, x[1].float())
  File "C:\Users\User\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\User\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "C:\Users\User\Anaconda3\envs\torch\lib\site-packages\torch\nn\functional.py", line 2009, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "C:\Users\User\Anaconda3\envs\torch\lib\site-packages\torch\nn\functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward

6. other points that help clarify my general problem:

I expect for the code to train and be able to predict. In other words, when I plug in a test example into the trained model:

pred = model(test).argmax(dim=1, keepdim=True)

the prediction should be a value from 0-9

Answer 1

It seems you need to pass a 1D LongTensor for the target. In your sample code, you passed a float value. I changed your sample code to work on MNIST dataset.

import torch
import torch.nn as nn

import os
from torchvision.datasets import MNIST
from torchvision import transforms

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(784, 99)
        self.output = nn.Linear(99, 10)

    def forward(self, x):
        x = self.hidden(x.view(x.size(0), -1))
        x = self.output(x)
        x = x.sigmoid()
        return x


train_loader = torch.utils.data.DataLoader(dataset=MNIST(os.getcwd()
                                          , train=True
                                          , transform=transforms.ToTensor()
                                          , download=True),
                                       batch_size=1,
                                       shuffle=True) 

n, target_size, num_epoc, learning_rate = 66, 1, 100, 0.001
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

target_value = torch.LongTensor([1])

for i in range(num_epoc):
    for x in train_loader:
        # run model and collect loss
        y = model.forward(x[0].float())
#         loss = criterion(y, x[1].float())
        loss = criterion(y, target_value)
        print(loss)
        break
    break

Answer 2

Since PyTorch version 1.10, nn.CrossEntropy() supports the so-called "soft’ (Using probabilistic) labels the only thing that you want to care about is that Input and Target has to have the same size.