3
$\begingroup$

I am trying to train a recurrent neural network that I built in keras on timeseries data to predict number of sales for next 10 days. For this, I've created my dataset as -

var(t) -> var(t+1)
var(t+1) -> var(t+2)
var(t+2) -> var(t+3)
var(t+3) -> var(t+4) and so on

I did Min-Max scaling on this data and the RNN code is as follows - 

model = Sequential()
model.add(LSTM(20, input_shape=(1, look_back),activation='tanh',bias_initializer='ones'))
model.add(Dense(1, activation='linear',bias_initializer='ones'))
opt=adam(lr=0.1)
model.compile(loss='mean_squared_error',optimizer=opt)
model.fit(xtrain, ytrain, epochs=100, batch_size=1, verbose=2)

But the plot I am getting is when I did predictions on xtrain (green = ytrain, blue = ypred) -

Observed vs expected signals

The rnn isn't learning anything at all. Its producing same results for each dataset. I've tried adding hidden layers, increasing number of neurons, changing parameters (learning rate, momentum), optimizers (sgd, adam, adagrad, rmsprop), lstm activation fxn (tanh, softsign). I got little fluctuations in some cases in the graph. But the output is mostly constant. Also, I've only 200 datasets.

Can someone please guide me what I am doing wrong here. What else I can try. Will small sized data not work using RNN at all ? If so, is there any other way to solve this problem (except ARIMA model) ?

EDIT - Increased batch size to 100 and epochs to 1000. Received some better results. Also, I did mean normalization [(x-mean)/std_dev] instead of MinMax scaling.

enter image description here

Improve this question
$\endgroup$
  • $\begingroup$ input shape is suspicious. what value as look_back? did you import Adam optimizer as such keras.optimizers.Adam as adam? learning rate of 0.1 is quite high. batch_size of 1 looks small. $\endgroup$ – tagoma Jul 14 '17 at 20:04
  • $\begingroup$ @edouard look_back is 1. yes, adam optimizer is from keras.optimizers.adam I've tried learning rate values - 0.01, 0.001, 0.03, 0.3 - none worked. $\endgroup$ – Ishan Jul 14 '17 at 20:12
  • $\begingroup$ @edouard I've edited the question with some changes that you mentioned. Decreasing learning rate to 0.01 didn't help. $\endgroup$ – Ishan Jul 14 '17 at 20:27
4
$\begingroup$

Try increasing your batch size. If your batch size is small, the gradients are a rough approximation of the true gradients.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.