0
$\begingroup$

I am using a dataset from kaggle to train a feed forward neural-neteork with no convolutional layers. I wanted to try it this was as a learning exercise with Pytorch without Transfer Learning and Convolutional Layers. Here is the code with its output.

Network Architecture

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()

        self.h0 = nn.Linear(99*99*3,1024)
        self.h1 = nn.Linear(1024,512)
        self.h2 = nn.Linear(512,256)
        self.h3 = nn.Linear(256,128)
        self.h4 = nn.Linear(128,1)

        self.dropout = nn.Dropout(p=0.2)

    def forward(self,x):
        x = x.view(x.shape[0],-1)
        x = torch.tanh(self.dropout(self.h0(x)))
        x = torch.tanh(self.dropout(self.h1(x)))
        x = torch.tanh(self.dropout(self.h2(x)))
        x = torch.tanh(self.dropout(self.h3(x)))
        x = torch.sigmoid(self.h4(x))

        return x

Paramerters

model = Classifier()
model.to(device)
criterion = nn.BCELoss()
optimizer = optim.SGD(model.parameters(),lr=0.00000001,momentum=0.9)

images,labels = next(iter(trainloader))
images,labels = make_gpu(images,labels)
print("Labels:{} ".format(labels.shape),"Images:{} ".format(images.shape))
probs = model(images)
print("Probabilities:{}".format(probs.shape))
loss = criterion(probs,labels)
loss.backward()
optimizer.step()
print(loss.item())
print(len(trainloader))
print(len(testloader))

Training and Validation

training_losses, testing_losses, test_acc, train_acc = [],[],[],[]

epochs = 10

for e in range(epochs):
    running_loss = 0
    tr_acc = 0
    for images,labels in trainloader:
        optimizer.zero_grad()
        images,labels = make_gpu(images,labels)
        probs = model(images)
        loss = criterion(probs,labels)
        loss.backward()
        optimizer.step()
        running_loss+=loss.item()
        probs = torch.round(probs)
        equals = probs == labels.view(*probs.shape)
        tr_acc += torch.mean(equals.type(torch.cuda.FloatTensor))



    else:
        testing_loss = 0
        acc = 0

        with torch.no_grad():
            model.eval()
            for images,labels in testloader:
                images,labels = make_gpu(images,labels)
                probs = model(images)
                loss = criterion(probs,labels)
                testing_loss+=loss
                probs = torch.round(probs)
                equals = probs == labels.view(*probs.shape)
                acc += torch.mean(equals.type(torch.cuda.FloatTensor))

            model.train()

            training_losses.append(running_loss/len(trainloader))
            testing_losses.append(testing_loss/len(testloader))
            test_acc.append(acc/len(testloader))
            train_acc.append(tr_acc/len(trainloader))

            print("Epoch: {}/{}.. ".format(e+1, epochs),
              "Training Loss: {:.3f}.. ".format(training_losses[-1]),
              "Test Loss: {:.3f}.. ".format(testing_losses[-1]),
              "Test Accuracy: {:.3f}..".format(test_acc[-1]),
              "Train Accuracy: {:.3f}".format(train_acc[-1]))

Output

Epoch: 1/10..  Training Loss: 0.694..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.503
Epoch: 2/10..  Training Loss: 0.694..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.502
Epoch: 3/10..  Training Loss: 0.694..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.501
Epoch: 4/10..  Training Loss: 0.694..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.503
Epoch: 5/10..  Training Loss: 0.694..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.503
Epoch: 6/10..  Training Loss: 0.695..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.497
Epoch: 7/10..  Training Loss: 0.695..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.499
Epoch: 8/10..  Training Loss: 0.695..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.499
Epoch: 9/10..  Training Loss: 0.695..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.499
Epoch: 10/10..  Training Loss: 0.695..  Test Loss: 0.694..  Test Accuracy: 0.505.. Train Accuracy: 0.501

Full Code

As you can see from the code the classifier performs worse the more it trains. Can someone please tell me why this is happening and how I can improve the model?

I have tried using SGD and Adam as the optimisers, both give a similar result. I have also tried learning rates 0.01 - 0.00000001 to no avail. Please help!

$\endgroup$
1
  • $\begingroup$ If you look at my full code, while loading the data if I had made shuffle=True in the trainloader the model starts showing normal behaviour. Any idea why this works? $\endgroup$ – Raikan 10 May 27 '20 at 17:26
0
$\begingroup$

That could be due to many reasons. One of the reasons could be gradient vanishing/explosion. Changing your nonlinear function could be a solution. For example, you can use Relu function instead of tanh function.

Also, using a early stopping rule could prevent this problem.

$\endgroup$
2
  • $\begingroup$ This kind of worked. thanks $\endgroup$ – Raikan 10 May 27 '20 at 17:24
  • $\begingroup$ I don't have the rep :/ $\endgroup$ – Raikan 10 Jun 18 '20 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.