I have a simple toy dataset for which the features have been encoded using a Encoder-Decoder NN.
I am using the hidden feature vector from the Encoder as the X input for training a 1-step lookahead model of the data.
I know the feature vector from the Encoder is good, because when I use this X as input into an vanilla out of the box LGBM tree model it fits almost perfectly to the Y output.
However, trying to train a simple 2 layer NN network (ReLU activation in first layer, Linear in second) which should learn the relatively easy mappings of the X feature vector to the Y output, it basically just learns to just average the output.
Now there is a reason why I want to use a NN model to predict the output here rather than just using the LGBM model, so why (if the LGBM model can do it) is my NN struggling so hard.
Below is the Prediction/Target from the LGBM Model:
And here is the same but from the NN model:
The reference Pytorch code for the NN model is:
alpha_hidden_1 = nn.Linear(encoder.hidden_size, hidden_alpha_size) alpha_out = nn.Linear(hidden_alpha_size, 1) x = F.relu(alpha_hidden_1(hidden_state_vector)) x = alpha_out(self.dropout(x))
I'm training it for 1000 epochs and have tried varying various hyperparameters to no avail.