0
$\begingroup$

I have text embeddings (768 dimensional vectors). I tried to build a feed forward neural network on classify the text into two classes.

The network I used.

class SimpleNet(nn.Module):
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(768, 84)
        self.fc2 = nn.Linear(84, 50)
        self.fc3 = nn.Linear(50, 2)
    
    def forward(self, x):
        x = x.view(-1, 768)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.softmax(self.fc3(x))
        return x

loss-function: torch.nn.CrossEntropyLoss()

optimizer: Adam

After training for 100 epochs I have the following plot for the train/val accuracy and loss

train-validation-accuracy-loss-plot

training epochs verbose output

trainval-loss-acc-simplenet-verbose-output-p1 trainval-loss-acc-simplenet-verbose-output-p2

My initial assumption was that the model reached its learning capacity and tried increasing the number of hidden layers, in the hope that a deeper model will probably be able to learn more

class SimpleNet(nn.Module):
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(768, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 128)
        self.fc4 = nn.Linear(128, 64)
        self.fc5 = nn.Linear(64, 2)
    
    def forward(self, x):
        x = x.view(-1, 768)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.softmax(self.fc5(x))
        return x

But the plots remain roughly same.

I tried tuning the learning rate by using lr_finder.
Also tried checking the results by changing the optimizer from Adam to SGD.
And also tried learning rate scheduler.
The results are pretty much same.
The accuracy didnot increase and the plots are similar.

Any ideas on what else I can try to increase the accuracy of the network?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.