I have a simple toy dataset for which the features have been encoded using a Encoder-Decoder NN.

I am using the hidden feature vector from the Encoder as the X input for training a 1-step lookahead model of the data.

I know the feature vector from the Encoder is good, because when I use this X as input into an vanilla out of the box LGBM tree model it fits almost perfectly to the Y output.

However, trying to train a simple 2 layer NN network (ReLU activation in first layer, Linear in second) which should learn the relatively easy mappings of the X feature vector to the Y output, it basically just learns to just average the output.

Now there is a reason why I want to use a NN model to predict the output here rather than just using the LGBM model, so why (if the LGBM model can do it) is my NN struggling so hard.

Below is the Prediction/Target from the LGBM Model:

enter image description here

And here is the same but from the NN model: enter image description here

The reference Pytorch code for the NN model is:

 alpha_hidden_1 = nn.Linear(encoder.hidden_size, hidden_alpha_size)
 alpha_out = nn.Linear(hidden_alpha_size, 1)

 x = F.relu(alpha_hidden_1(hidden_state_vector))
 x = alpha_out(self.dropout(x))

I'm training it for 1000 epochs and have tried varying various hyperparameters to no avail.


1 Answer 1


It looks like your neural network has a high bias. There are a couple of things you can do as a first step to attack this:

  • increase number of hidden layers
  • increase number of neurons
  • increase number of training epochs

Generally, neural networks have a lot of hyperparameters and so would be good to optimise using a gridsearch for example.

In terms of your first two models, it is of note that you need to double check they are not overfitting the training set.

  • $\begingroup$ The NN not learning hints to the learning rate being too big ... I would suggest to add one or two 0 to the learning rate. $\endgroup$ Commented Feb 10, 2023 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.