2
$\begingroup$

Playing around with weather data, I have set up a simple RNN with one layer of GRUs. It is trained to recover the temperature of the next day, given weather data of the last 5 days, each with 1-hour intervals.

What I find peculiar is that after training several epochs, the result is something that has a lot of the small scale features of the data, but it lacks the large-scale structure. Frequently, there seems to be just an offset between prediction and data, e.g. at x=900 below. On the other hand, the very steep peaks are also not well fitted.

Here is the code of the model:

model = keras.Sequential()
model.add(keras.layers.GRU(units=120,
                         activation='relu',
                         dropout=0.1,
                         recurrent_dropout=0.4,
                         return_sequences=False,
                         input_shape=(120, 14)))
model.add(keras.layers.Dense(1))
model.compile(optimizer=keras.optimizers.RMSprop(),loss='mse')

The training set has 120 (=5 days) of weather data, with 15 variables at each point in time. For example, these are the first 2 of 120 vectors in the first training set:

print (training_x[0,0:2,:])
[[ 0.87422976 -2.0740129  -2.12744145 -2.05861548  1.04950092 -1.32397418
  -1.53525603 -0.78058659 -1.53697269 -1.53946235  2.29360559 -0.01027133
  -0.01893096 -0.25892163 -1.72366227]
 [ 0.87183698 -2.02652589 -2.07923146 -1.96947126  1.14054157 -1.30976048
  -1.49940654 -0.78875511 -1.49932401 -1.50404649  2.24182343 -0.02325901
  -0.03515888 -0.09510368 -1.72366227]]

They were normalized beforehand.

(I should note it is based on this)

I am trying to figure out whether this is a well-known phenomenon and/or in which direction I should change my model in order to compensate. I am using 120 GRU 'units' followed by a single unit dense layer.

Example data

$\endgroup$
  • $\begingroup$ Please, paste the code of your model, it'd be very useful $\endgroup$ – Alber8295 Oct 30 '18 at 14:11
  • 1
    $\begingroup$ I added the code generating the model! $\endgroup$ – rugermini Oct 30 '18 at 14:24
  • $\begingroup$ This is a very interesting question. I verify having the same behavior when I use GRU RNN for a problem of mine, particulary a single-layer GRU RNN for 1-step-ahead prediction given the previous 30 steps. I am using 10 neurons. I haven't reached a verdict yet, but using Bidirectional GRU neurons seems to resolve the issue for me. BUT I am more than interested in solving it for plain GRU units. $\endgroup$ – pcko1 Oct 30 '18 at 14:26
  • $\begingroup$ Thanks! Post also a few (just 2 or 3) samples of your training dataset. $\endgroup$ – Alber8295 Oct 30 '18 at 14:34
  • 1
    $\begingroup$ Yes, of course... Added them above. $\endgroup$ – rugermini Oct 30 '18 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.