I am using Ensemble PyTorch to train a voting classifier. My dataset includes around 60k records. I trained a Neural Network with Cross-entropy loss. Below is my model architecture
class Classifier(nn.Module): def __init__(self): super(Classifier, self).__init__() self.linear1 = nn.Linear(10, 128) self.linear2 = nn.Linear(128, 128) self.linear3 = nn.Linear(128, 60) def forward(self, data): data = data.view(data.size(0), -1) output = F.relu(self.linear1(data)) output = F.relu(self.linear2(output)) output = self.linear3(output) return output
I am using Adam optimizer, CosineAnnealingLR scheduler, number of estimators for ensemble is 10, batch size is 100 and I am running it for 150 epochs
criterion = nn.CrossEntropyLoss() model.set_criterion(criterion) model.set_optimizer('Adam', lr=1e-3, weight_decay=5e-4) model.set_scheduler( "CosineAnnealingLR", T_max=epochs, )
I am getting below learning curves for training/validation loss and accuracy. The validation loss is very high but it is decreasing very slowly and it is much much higher than training loss. I suspect that it is overfitting and that is why the validation loss is too high. Is it indeed the case of overfitting or underfitting(I am confused here as the validation accuracy is quite good and it is not deviating too much from the training accuracy). Please guide me on how can I reduce the validation loss and if you can please provide any pointers on improving the model.