NN Making Poor Averaging Fit whilst LGBM Regressor Fits Perfectly

Question

I have a simple toy dataset for which the features have been encoded using a Encoder-Decoder NN.

I am using the hidden feature vector from the Encoder as the X input for training a 1-step lookahead model of the data.

I know the feature vector from the Encoder is good, because when I use this X as input into an vanilla out of the box LGBM tree model it fits almost perfectly to the Y output.

However, trying to train a simple 2 layer NN network (ReLU activation in first layer, Linear in second) which should learn the relatively easy mappings of the X feature vector to the Y output, it basically just learns to just average the output.

Now there is a reason why I want to use a NN model to predict the output here rather than just using the LGBM model, so why (if the LGBM model can do it) is my NN struggling so hard.

Below is the Prediction/Target from the LGBM Model:

And here is the same but from the NN model:

The reference Pytorch code for the NN model is:

 alpha_hidden_1 = nn.Linear(encoder.hidden_size, hidden_alpha_size)
 alpha_out = nn.Linear(hidden_alpha_size, 1)

 x = F.relu(alpha_hidden_1(hidden_state_vector))
 x = alpha_out(self.dropout(x))

I'm training it for 1000 epochs and have tried varying various hyperparameters to no avail.

Answer 1

It looks like your neural network has a high bias. There are a couple of things you can do as a first step to attack this:

• increase number of hidden layers
• increase number of neurons
• increase number of training epochs

Generally, neural networks have a lot of hyperparameters and so would be good to optimise using a gridsearch for example.

In terms of your first two models, it is of note that you need to double check they are not overfitting the training set.