I trained a GRU model on some data and then created a bunch of predictions on a test set.

The predictions are really bad, as indicated by a near zero R2 score.

I notice that the variance of the model predictions are much smaller than the actual training data. i.e it seems like the model is overfit to the mean:

enter image description here

But why is this? I made sure to stop training/use hyperparameters where there was no model overfitting, so why are model predictions centred around the mean and less dispersed than the actual variance of the data set?

My model, if it is relevant:

model = Sequential()
model.add(GRU(100, activation='relu', input_shape=(3, 280), recurrent_dropout = 0.2, dropout = 0.2))
model.compile(optimizer='adam', loss='mse', metrics=['mae'])

# fit model
history = model.fit(X_train, y_train, epochs=40, verbose=1, validation_split=0.33)

The first thing you can do here is to scale your output data before training the model. For instance

$$ \dfrac{x_i - \bar{x}_{train}}{\sigma_{x_{train}}} $$

If that does not work, then you can have a look at the model and / or training parameters.

  • $\begingroup$ Yes, I normalised the variables before training. $\endgroup$
    – eartoolbox
    Aug 22 '20 at 22:04
  • $\begingroup$ then something is off, if you used some sort of scaling before training the variance should be lower. What scaling did you use? $\endgroup$ Aug 23 '20 at 0:07
  • $\begingroup$ The variance is lower. I used the same equation you presented to normalise the variables. $\endgroup$
    – eartoolbox
    Aug 23 '20 at 0:38
  • $\begingroup$ So, it just occurred to me that I need to un-normalise the predictions right? $\endgroup$
    – eartoolbox
    Aug 23 '20 at 10:07
  • $\begingroup$ yes - in order to avoid bias you need to estimate the normalisation parameters on the training set (save them somewhere) then you can use these values to un-normalise the prediction $\endgroup$ Aug 23 '20 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.