# Training and validation accuracy stagnating after a few epochs for text embeddings

I have text embeddings (768 dimensional vectors). I tried to build a feed forward neural network on classify the text into two classes.

The network I used.

class SimpleNet(nn.Module):
def __init__(self):
super(SimpleNet, self).__init__()
self.fc1 = nn.Linear(768, 84)
self.fc2 = nn.Linear(84, 50)
self.fc3 = nn.Linear(50, 2)

def forward(self, x):
x = x.view(-1, 768)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = F.softmax(self.fc3(x))
return x


loss-function: torch.nn.CrossEntropyLoss()

After training for 100 epochs I have the following plot for the train/val accuracy and loss

training epochs verbose output

My initial assumption was that the model reached its learning capacity and tried increasing the number of hidden layers, in the hope that a deeper model will probably be able to learn more

class SimpleNet(nn.Module):
def __init__(self):
super(SimpleNet, self).__init__()
self.fc1 = nn.Linear(768, 512)
self.fc2 = nn.Linear(512, 256)
self.fc3 = nn.Linear(256, 128)
self.fc4 = nn.Linear(128, 64)
self.fc5 = nn.Linear(64, 2)

def forward(self, x):
x = x.view(-1, 768)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = F.relu(self.fc3(x))
x = F.relu(self.fc4(x))
x = F.softmax(self.fc5(x))
return x


But the plots remain roughly same.

I tried tuning the learning rate by using lr_finder.
Also tried checking the results by changing the optimizer from Adam to SGD.
And also tried learning rate scheduler.
The results are pretty much same.
The accuracy didnot increase and the plots are similar.

Any ideas on what else I can try to increase the accuracy of the network?